feat(bench): serving benchmark — the baseline instrument for the migration

A scripted 8-turn conversation through the real pipeline (tool call,
memory recall, librarian route, small talk), run against any base URL so
the same instrument measures Ollama today and llama-server after the
cutover. Non-stream mode records wall time and usage; --stream records
time-to-first-token; a VRAM poller and an optional interleaved second
tenant capture the contention the serving plan calls out.

Planted facts carry a BENCH-MARKER prefix so anything the biographer
learns from a bench run stays recognizable in the memory stores.

First smoke against production: 9.7s mean per simple turn, and TTFT
equals wall time — the client sees nothing until the whole
steward-orchestrate-synthesize pipeline has finished. Time-to-first-word
for desklock is currently the full pipeline latency, which makes
streamed synthesis a first-class goal of the serving work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-11 21:07:57 +02:00
co-authored by Claude Fable 5
parent 014afc960c
commit f578ebded8
3 changed files with 366 additions and 1 deletions
+4 -1
View File
@@ -1,4 +1,4 @@
.PHONY: help setup run test test-unit test-integration test-contracts lint typecheck clean
.PHONY: help setup run test test-unit test-integration test-contracts bench-serving lint typecheck clean
VENV := .venv
PYTHON := $(VENV)/bin/python
@@ -50,6 +50,9 @@ test-integration: ## Run integration tests (needs Claude/Ollama)
test-contracts: ## Wire-level contract tests against live service boundaries
$(PYTEST) tests/contracts -v --no-cov
bench-serving: ## Serving benchmark vs a live backend (BENCH_ARGS="--base-url ... --stream")
$(PYTHON) scripts/bench_serving.py $(BENCH_ARGS)
lint: ## Run ruff linter and formatter check
$(RUFF) check src tests
$(RUFF) format --check src tests