Extracts the residency snapshot/restore into scripts/ollama_residency.py
so the two benchmarks cannot drift, and applies it to
benchmark_tool_calling.py, which had no protection at all.
That script was the more dangerous of the two. It rewrites
OLLAMA_DEFAULT_MODEL in .env and lets uvicorn reload onto it, restoring
the original only after the loop — so any crash or interrupt left the
*running server* pointed at the benchmark model. Its DEFAULT_MODELS
begins with mistral-nemo-large, the 9.2G model implicated in the
2026-08-07 VRAM outage. Both the .env restore and the residency restore
now run from `finally`.
SIGTERM is handled explicitly in the shared module. Python runs `finally`
for SIGINT, which arrives as KeyboardInterrupt, but the default SIGTERM
action terminates outright, so `timeout` or a plain `kill` skipped the
guard entirely.
Co-Authored-By: Claude <noreply@anthropic.com>
Benchmarking swaps models on the GPU production is serving from. Ollama
evicts to make room, so the first run unpinned gemma4:e2b and left
gemma4:e4b resident: the next voice turn would have paid a ~36s cold
load, and only the monitoring noticing unexpected_models caught it.
Snapshot residency and pinning before the run, then evict whatever the
benchmark loaded and re-pin what was pinned before.
The restore is wired to SIGTERM as well as the normal exit path. Python
runs `finally` for SIGINT, which arrives as KeyboardInterrupt, but the
default SIGTERM action terminates outright — so a `timeout`, a systemd
stop or a plain `kill` skipped the guard entirely. That was not
theoretical: the first SIGTERM after adding this bypassed it, and the
pinned model survived only because the run had not reached the second
model yet.
Co-Authored-By: Claude <noreply@anthropic.com>
Measures Steward routing against model and thinking settings by talking
to Ollama directly. No server, no agents, nothing executed — the
mutating fixtures only ever produce a routing decision — so the run is
cheap, repeatable and isolates routing from everything downstream. The
request body mirrors StewardAgent._call_ollama, so the `unset` cell is
exactly what production sends today.
Three thinking settings rather than two. `unset` is production, and it
is not neutral: gemma4 reasons by default and returns no `thinking`
field, so those tokens are generated and discarded.
Scoring is asymmetric on purpose. Each fixture carries `forbid` as well
as `expect`, because over-routing is the predicted failure when thinking
is off and it is the expensive one — a spurious librarian is a real web
call on a query that asked for arithmetic.
The adversarial group is regression coverage for the extraction fix in
a905363: those queries invite the vocabulary that used to select agents
by substring, so they now assert that routing follows what the Steward
decided rather than the words it used while explaining.
Co-Authored-By: Claude <noreply@anthropic.com>
jpmschweitzerandClaude