feat(backend): BACKEND_SLOT_PINNING — per-phase engine slot ownership
Each pipeline phase owns one llama-server slot (steward 0, orchestrator 1, synthesizer 2), carried as id_slot in extra_body through the same mechanism tool_choice already uses, so the phase's stable prompt prefix stays in that slot's KV cache and a turn re-prefills only its new tokens. Off by default; a no-op on the Claude backend and ignored by Ollama, so the flag is safe on any backend and the cutover itself stays a pure env swap. The merge helper preserves existing extra_body keys — mutation-checked (dropping the merge fails exactly the test written for it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -162,15 +162,20 @@ class StewardAgent:
|
||||
OpenAI-compatible server (Ollama, llama-server) can sit behind
|
||||
OLLAMA_HOST without this method knowing which.
|
||||
"""
|
||||
payload: dict = {
|
||||
"model": self.ollama_model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": 0.3, # Lower = more consistent
|
||||
"top_p": 0.9,
|
||||
}
|
||||
if config.BACKEND_SLOT_PINNING:
|
||||
# The steward owns engine slot 0 (see BACKEND_SLOT_PINNING)
|
||||
payload["id_slot"] = 0
|
||||
|
||||
async with httpx.AsyncClient(timeout=self.timeout) as client:
|
||||
response = await client.post(
|
||||
f"{self.ollama_host}/v1/chat/completions",
|
||||
json={
|
||||
"model": self.ollama_model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": 0.3, # Lower = more consistent
|
||||
"top_p": 0.9,
|
||||
},
|
||||
json=payload,
|
||||
)
|
||||
|
||||
response.raise_for_status()
|
||||
|
||||
Reference in New Issue
Block a user