tatlock/docs/steward-routing-findings.md

Steward Routing & Thinking — Findings

Outcome: no change shipped. The Steward stays on gemma4:e2b with model thinking left at its default (on). Every alternative was measured and every one loses. This document exists so the experiment is not repeated on the same premise.

Run 2026-08-08 with scripts/benchmark_routing.py and scripts/fixtures/routing_fixtures.py (40 labelled queries, one repeat per cell, temperature 0.3 as production sends).

---

The premise was wrong

The experiment was designed around an observation that the Steward pays ~300 tokens per turn for reasoning that is generated and thrown away: it calls /api/generate, gemma4 reasons by default, and no thinking field comes back in the response. Disabling thinking therefore looked close to free.

It is not. The reasoning is not discarded — it is emitted inline in response, and it is what produces a correct DELEGATE: line. Those tokens are the work, not waste. Suppressing them costs 12.5 points of routing accuracy.

Results

config	exact	under	over	tokens	latency	resident	predicted	co-resident with nomic
e2b, thinking (production)	97.5%	2.5%	0%	361	5179 ms	1778 MB	7.8 GiB	yes
e2b, think: false	85.0%	12.5%	5.0%	48	1435 ms	1778 MB	7.8 GiB	yes
e4b, thinking	100%	0%	0%	192	4726 ms	3089 MB	10.6 GiB	no
e4b, think: false	97.5%	2.5%	0%	52	2269 ms	3089 MB	10.6 GiB	no

think: true was also measured and landed within one fixture of the default on both models, so production's implicit thinking is the same thing as asking for it explicitly. Format compliance was 100% in every cell — a DELEGATE: line is always emitted.

With 40 fixtures and one repeat, each result is worth 2.5 points, so the 97.5-vs-100 gaps are single fixtures and inside the noise. The latency and token medians (40 calls each) and the e2b think: false degradation (6 failures with a consistent mechanism) are the parts worth trusting.

Why each alternative loses

think: false on e2b — 85% exact, and the failures are not random. All three multi-capability fixtures under-route, each missing a second capability. Without reasoning the model names one capability and stops decomposing. It is not degraded across the board; it specifically stops handling compound requests, which is where a user would most notice the Butler quietly doing half the job.

e4b, either setting — disqualified by memory, not by quality. Ollama predicts 10.6 GiB for it at 16k context. Maximum available on this card is ~7.9 GiB (10.4 free − 2.0 GPU overhead − 0.46 minimum), so e4b always exceeds the budget and evicts every co-resident before loading. Observed directly: loading it threw out both gemma4:e2b and nomic-embed-text. Losing nomic means Tatlock memory and library-desk thrash on every embedding call. Note this is not caused by the 2 GiB reservation — without it, available would be ~9.7 GiB, still under 10.6.

Lower OLLAMA_CONTEXT_LENGTH — the obvious way to free headroom, and it does not work. Dropping 16384 → 2048, an 8× reduction, moved the prediction only from 7.8 to 6.7 GiB. The prediction is dominated by weights and batch size, not KV cache. It would also truncate the Librarian's retrieved passages and webber's code context for a 14% saving that funds nothing.

Per-request num_ctx — worse. A single request with a different num_ctx reloads the shared runner, which drops the keep_alive: -1 pin (expiry fell from year-2318 to a 2-hour default) and evicts nomic. Three services share this Ollama, so mixed context sizes are a thrash generator, and it fails silently.

OLLAMA_NUM_PARALLEL > 1 — never viable here. e2b already predicts 7.8 GiB against ~7.9 available, so there is no room for a second slot at any context length. It is also set to 1 deliberately, to avoid batch overflow panics.

What the two axes actually control

They do not interact, which is the useful part:

• Model choice governs VRAM and co-residency. e2b 1778 MB, e4b 3089 MB.
• Think setting governs tokens, latency and routing quality — and costs nothing in VRAM. Verified: e2b is resident at 1778 MB with think unset, true and false alike, because the KV cache is allocated for the full context at load time and think is a per-request generation parameter.

So the only real question is whether 313 tokens and 3.7 seconds are worth 12.5 points of compound-query routing. On a turn that is already three sequential Ollama calls, they are.

Prerequisite: the extraction fix

These numbers are only meaningful because _extract_capabilities was fixed first (commit a905363). It previously substring-matched capability domains across the Steward's entire response, so ordinary English in the REASON: line selected agents — "description" contains the housekeeper domain "script", "acknowledge" contains "knowledge" and "know".

That made prose length a routing input. Benchmarking against it would have shown think: false improving routing purely because shorter output produces fewer accidental substring hits — a thinking policy derived from a parsing artefact. The adversarial fixture group is regression coverage for exactly this.

If this is revisited

The constraint is the single 11 GB card, not the model. A second inference host (forge) removes it entirely, and e4b's 100% routing becomes reachable without evicting anything. Re-run then; on this card the answer is settled.

scripts/benchmark_routing.py takes --models, --think and --repeats, and restores GPU residency on exit — including on SIGTERM, which the first version did not.