diff --git a/docs/steward-routing-findings.md b/docs/steward-routing-findings.md new file mode 100644 index 0000000..0c48255 --- /dev/null +++ b/docs/steward-routing-findings.md @@ -0,0 +1,110 @@ +# Steward Routing & Thinking — Findings + +**Outcome: no change shipped.** The Steward stays on `gemma4:e2b` with model +thinking left at its default (on). Every alternative was measured and every one +loses. This document exists so the experiment is not repeated on the same +premise. + +Run 2026-08-08 with `scripts/benchmark_routing.py` and +`scripts/fixtures/routing_fixtures.py` (40 labelled queries, one repeat per +cell, temperature 0.3 as production sends). + +--- + +## The premise was wrong + +The experiment was designed around an observation that the Steward pays ~300 +tokens per turn for reasoning that is generated and thrown away: it calls +`/api/generate`, gemma4 reasons by default, and **no `thinking` field comes back +in the response**. Disabling thinking therefore looked close to free. + +It is not. The reasoning is not discarded — it is emitted inline in `response`, +and it is what produces a correct `DELEGATE:` line. Those tokens are the work, +not waste. Suppressing them costs 12.5 points of routing accuracy. + +## Results + +| config | exact | under | over | tokens | latency | resident | predicted | co-resident with nomic | +|---|---|---|---|---|---|---|---|---| +| **e2b, thinking** *(production)* | **97.5%** | 2.5% | 0% | 361 | 5179 ms | 1778 MB | 7.8 GiB | yes | +| e2b, `think: false` | 85.0% | 12.5% | 5.0% | 48 | 1435 ms | 1778 MB | 7.8 GiB | yes | +| e4b, thinking | 100% | 0% | 0% | 192 | 4726 ms | 3089 MB | 10.6 GiB | **no** | +| e4b, `think: false` | 97.5% | 2.5% | 0% | 52 | 2269 ms | 3089 MB | 10.6 GiB | **no** | + +`think: true` was also measured and landed within one fixture of the default on +both models, so production's implicit thinking is the same thing as asking for +it explicitly. Format compliance was 100% in every cell — a `DELEGATE:` line is +always emitted. + +With 40 fixtures and one repeat, each result is worth 2.5 points, so the +97.5-vs-100 gaps are single fixtures and inside the noise. The latency and token +medians (40 calls each) and the e2b `think: false` degradation (6 failures with a +consistent mechanism) are the parts worth trusting. + +## Why each alternative loses + +**`think: false` on e2b** — 85% exact, and the failures are not random. All three +multi-capability fixtures under-route, each missing a second capability. Without +reasoning the model names one capability and stops decomposing. It is not +degraded across the board; it specifically stops handling compound requests, +which is where a user would most notice the Butler quietly doing half the job. + +**e4b, either setting** — disqualified by memory, not by quality. Ollama predicts +**10.6 GiB** for it at 16k context. Maximum available on this card is ~7.9 GiB +(10.4 free − 2.0 GPU overhead − 0.46 minimum), so e4b *always* exceeds the budget +and evicts every co-resident before loading. Observed directly: loading it threw +out both `gemma4:e2b` and `nomic-embed-text`. Losing nomic means Tatlock memory +and library-desk thrash on every embedding call. Note this is not caused by the +2 GiB reservation — without it, available would be ~9.7 GiB, still under 10.6. + +**Lower `OLLAMA_CONTEXT_LENGTH`** — the obvious way to free headroom, and it does +not work. Dropping 16384 → 2048, an 8× reduction, moved the prediction only from +7.8 to 6.7 GiB. The prediction is dominated by weights and batch size, not KV +cache. It would also truncate the Librarian's retrieved passages and webber's +code context for a 14% saving that funds nothing. + +**Per-request `num_ctx`** — worse. A single request with a different `num_ctx` +reloads the shared runner, which **drops the `keep_alive: -1` pin** (expiry fell +from year-2318 to a 2-hour default) and evicts nomic. Three services share this +Ollama, so mixed context sizes are a thrash generator, and it fails silently. + +**`OLLAMA_NUM_PARALLEL > 1`** — never viable here. e2b already predicts 7.8 GiB +against ~7.9 available, so there is no room for a second slot at any context +length. It is also set to 1 deliberately, to avoid batch overflow panics. + +## What the two axes actually control + +They do not interact, which is the useful part: + +- **Model choice** governs VRAM and co-residency. e2b 1778 MB, e4b 3089 MB. +- **Think setting** governs tokens, latency and routing quality — and costs + **nothing** in VRAM. Verified: e2b is resident at 1778 MB with `think` unset, + true and false alike, because the KV cache is allocated for the full context at + load time and `think` is a per-request generation parameter. + +So the only real question is whether 313 tokens and 3.7 seconds are worth 12.5 +points of compound-query routing. On a turn that is already three sequential +Ollama calls, they are. + +## Prerequisite: the extraction fix + +These numbers are only meaningful because `_extract_capabilities` was fixed first +(commit `a905363`). It previously substring-matched capability *domains* across +the Steward's entire response, so ordinary English in the `REASON:` line selected +agents — "description" contains the housekeeper domain "script", "acknowledge" +contains "knowledge" and "know". + +That made **prose length a routing input**. Benchmarking against it would have +shown `think: false` improving routing purely because shorter output produces +fewer accidental substring hits — a thinking policy derived from a parsing +artefact. The `adversarial` fixture group is regression coverage for exactly this. + +## If this is revisited + +The constraint is the single 11 GB card, not the model. A second inference host +(*forge*) removes it entirely, and e4b's 100% routing becomes reachable without +evicting anything. Re-run then; on this card the answer is settled. + +`scripts/benchmark_routing.py` takes `--models`, `--think` and `--repeats`, and +restores GPU residency on exit — including on SIGTERM, which the first version +did not.