# Steward Routing & Thinking — Findings

**Outcome: no change shipped.** The Steward stays on `gemma4:e2b` with model
thinking left at its default (on). Every alternative was measured and every one
loses. This document exists so the experiment is not repeated on the same
premise.

Run 2026-08-08 with `scripts/benchmark_routing.py` and
`scripts/fixtures/routing_fixtures.py` (40 labelled queries, one repeat per
cell, temperature 0.3 as production sends).

---

## The premise was wrong

The experiment was designed around an observation that the Steward pays ~300
tokens per turn for reasoning that is generated and thrown away: it calls
`/api/generate`, gemma4 reasons by default, and **no `thinking` field comes back
in the response**. Disabling thinking therefore looked close to free.

It is not. The reasoning is not discarded — it is emitted inline in `response`,
and it is what produces a correct `DELEGATE:` line. Those tokens are the work,
not waste. Suppressing them costs 12.5 points of routing accuracy.

## Results

| config | exact | under | over | tokens | latency | resident | predicted | co-resident with nomic |
|---|---|---|---|---|---|---|---|---|
| **e2b, thinking** *(production)* | **97.5%** | 2.5% | 0% | 361 | 5179 ms | 1778 MB | 7.8 GiB | yes |
| e2b, `think: false` | 85.0% | 12.5% | 5.0% | 48 | 1435 ms | 1778 MB | 7.8 GiB | yes |
| e4b, thinking | 100% | 0% | 0% | 192 | 4726 ms | 3089 MB | 10.6 GiB | **no** |
| e4b, `think: false` | 97.5% | 2.5% | 0% | 52 | 2269 ms | 3089 MB | 10.6 GiB | **no** |

`think: true` was also measured and landed within one fixture of the default on
both models, so production's implicit thinking is the same thing as asking for
it explicitly. Format compliance was 100% in every cell — a `DELEGATE:` line is
always emitted.

With 40 fixtures and one repeat, each result is worth 2.5 points, so the
97.5-vs-100 gaps are single fixtures and inside the noise. The latency and token
medians (40 calls each) and the e2b `think: false` degradation (6 failures with a
consistent mechanism) are the parts worth trusting.

## Why each alternative loses

**`think: false` on e2b** — 85% exact, and the failures are not random. All three
multi-capability fixtures under-route, each missing a second capability. Without
reasoning the model names one capability and stops decomposing. It is not
degraded across the board; it specifically stops handling compound requests,
which is where a user would most notice the Butler quietly doing half the job.

**e4b, either setting** — disqualified by memory, not by quality. Ollama predicts
**10.6 GiB** for it at 16k context. Maximum available on this card is ~7.9 GiB
(10.4 free − 2.0 GPU overhead − 0.46 minimum), so e4b *always* exceeds the budget
and evicts every co-resident before loading. Observed directly: loading it threw
out both `gemma4:e2b` and `nomic-embed-text`. Losing nomic means Tatlock memory
and library-desk thrash on every embedding call. Note this is not caused by the
2 GiB reservation — without it, available would be ~9.7 GiB, still under 10.6.

**Lower `OLLAMA_CONTEXT_LENGTH`** — the obvious way to free headroom, and it does
not work. Dropping 16384 → 2048, an 8× reduction, moved the prediction only from
7.8 to 6.7 GiB. The prediction is dominated by weights and batch size, not KV
cache. It would also truncate the Librarian's retrieved passages and webber's
code context for a 14% saving that funds nothing.

**Per-request `num_ctx`** — worse. A single request with a different `num_ctx`
reloads the shared runner, which **drops the `keep_alive: -1` pin** (expiry fell
from year-2318 to a 2-hour default) and evicts nomic. Three services share this
Ollama, so mixed context sizes are a thrash generator, and it fails silently.

**`OLLAMA_NUM_PARALLEL > 1`** — never viable here. e2b already predicts 7.8 GiB
against ~7.9 available, so there is no room for a second slot at any context
length. It is also set to 1 deliberately, to avoid batch overflow panics.

## What the two axes actually control

They do not interact, which is the useful part:

- **Model choice** governs VRAM and co-residency. e2b 1778 MB, e4b 3089 MB.
- **Think setting** governs tokens, latency and routing quality — and costs
  **nothing** in VRAM. Verified: e2b is resident at 1778 MB with `think` unset,
  true and false alike, because the KV cache is allocated for the full context at
  load time and `think` is a per-request generation parameter.

So the only real question is whether 313 tokens and 3.7 seconds are worth 12.5
points of compound-query routing. On a turn that is already three sequential
Ollama calls, they are.

## Prerequisite: the extraction fix

These numbers are only meaningful because `_extract_capabilities` was fixed first
(commit `a905363`). It previously substring-matched capability *domains* across
the Steward's entire response, so ordinary English in the `REASON:` line selected
agents — "description" contains the housekeeper domain "script", "acknowledge"
contains "knowledge" and "know".

That made **prose length a routing input**. Benchmarking against it would have
shown `think: false` improving routing purely because shorter output produces
fewer accidental substring hits — a thinking policy derived from a parsing
artefact. The `adversarial` fixture group is regression coverage for exactly this.

## If this is revisited

The constraint is the single 11 GB card, not the model. A second inference host
(*forge*) removes it entirely, and e4b's 100% routing becomes reachable without
evicting anything. Re-run then; on this card the answer is settled.

`scripts/benchmark_routing.py` takes `--models`, `--think` and `--repeats`, and
restores GPU residency on exit — including on SIGTERM, which the first version
did not.