Benchmarking swaps models on the GPU production is serving from. Ollama
evicts to make room, so the first run unpinned gemma4:e2b and left
gemma4:e4b resident: the next voice turn would have paid a ~36s cold
load, and only the monitoring noticing unexpected_models caught it.
Snapshot residency and pinning before the run, then evict whatever the
benchmark loaded and re-pin what was pinned before.
The restore is wired to SIGTERM as well as the normal exit path. Python
runs `finally` for SIGINT, which arrives as KeyboardInterrupt, but the
default SIGTERM action terminates outright — so a `timeout`, a systemd
stop or a plain `kill` skipped the guard entirely. That was not
theoretical: the first SIGTERM after adding this bypassed it, and the
pinned model survived only because the run had not reached the second
model yet.
Co-Authored-By: Claude <noreply@anthropic.com>
Measures Steward routing against model and thinking settings by talking
to Ollama directly. No server, no agents, nothing executed — the
mutating fixtures only ever produce a routing decision — so the run is
cheap, repeatable and isolates routing from everything downstream. The
request body mirrors StewardAgent._call_ollama, so the `unset` cell is
exactly what production sends today.
Three thinking settings rather than two. `unset` is production, and it
is not neutral: gemma4 reasons by default and returns no `thinking`
field, so those tokens are generated and discarded.
Scoring is asymmetric on purpose. Each fixture carries `forbid` as well
as `expect`, because over-routing is the predicted failure when thinking
is off and it is the expensive one — a spurious librarian is a real web
call on a query that asked for arithmetic.
The adversarial group is regression coverage for the extraction fix in
a905363: those queries invite the vocabulary that used to select agents
by substring, so they now assert that routing follows what the Steward
decided rather than the words it used while explaining.
Co-Authored-By: Claude <noreply@anthropic.com>
Three migrations left their documentation behind:
wakeup.sh was replaced by the Makefile during the project structure
consolidation, but AGENTS.md and the e2e README still tell you to run it.
The log path moved to build/logs/server.log at the same time.
The local model moved to gemma4:e2b, but the e2e prerequisites and the
benchmark recommendation still name mistral-nemo.
The benchmark figures in CLAUDE.md predate the current model. Measured
2026-08-07: ~95 tok/s, full flow ~10-13s for simple turns, cold model load
~36s rather than ~8s. A turn costs three sequential Ollama calls and ~710
generated tokens regardless of how trivial the question is.
Co-Authored-By: Claude <noreply@anthropic.com>
The homelab is retiring *.schweitz.internal and will rebind host
ports to loopback; container-to-container traffic must use container
names on docker-dataplane.
- SEARXNG_HOST: http://localhost:8087 -> http://searxng:8080
(SearXNG's internal port is 8080; 8087 was the host-published port)
- LIBRARY_DESK_HOST: http://localhost:8089 -> http://library-desk:8089
- CORE_API_HOST: http://localhost:8090 -> http://core-api:8083
(8090 is the Scheduler's host port; Core-API serves 8083 internally,
confirmed by the housekeeper client and test suite hitting :8083)
- scripts/test_housekeeper.sh: reach Core-API via localhost:8083
instead of the LAN IP, which will refuse after loopback rebinding
Local development against host-published ports keeps working via .env
overrides (.env.example unchanged; localhost stays valid on the host).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gemma4:e2b has native function calling with dedicated tool tokens,
achieving 100% tool selection accuracy in benchmarks vs 67% for
mistral-nemo-large, with 5-8x faster response times (2-4s vs 15-20s)
and lower VRAM usage (8GB vs 9.2GB).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rewrite system prompt with negative constraints and step-by-step process
- Set temperature to 0.1 for deterministic tool calling
- Sort room groups to top of device list (address positional bias)
- Add [ROOM GROUP] marker in list_devices output
- Update tool docstrings with explicit entity_id= parameter examples
- Add optimization findings doc (experiment log: 0% → 100% success)
- Add test script for room group detection regression testing
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
jpmschweitzerandClaude