feat(bench): serving benchmark — the baseline instrument for the migration
A scripted 8-turn conversation through the real pipeline (tool call, memory recall, librarian route, small talk), run against any base URL so the same instrument measures Ollama today and llama-server after the cutover. Non-stream mode records wall time and usage; --stream records time-to-first-token; a VRAM poller and an optional interleaved second tenant capture the contention the serving plan calls out. Planted facts carry a BENCH-MARKER prefix so anything the biographer learns from a bench run stays recognizable in the memory stores. First smoke against production: 9.7s mean per simple turn, and TTFT equals wall time — the client sees nothing until the whole steward-orchestrate-synthesize pipeline has finished. Time-to-first-word for desklock is currently the full pipeline latency, which makes streamed synthesis a first-class goal of the serving work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
.PHONY: help setup run test test-unit test-integration test-contracts lint typecheck clean
|
||||
.PHONY: help setup run test test-unit test-integration test-contracts bench-serving lint typecheck clean
|
||||
|
||||
VENV := .venv
|
||||
PYTHON := $(VENV)/bin/python
|
||||
@@ -50,6 +50,9 @@ test-integration: ## Run integration tests (needs Claude/Ollama)
|
||||
test-contracts: ## Wire-level contract tests against live service boundaries
|
||||
$(PYTEST) tests/contracts -v --no-cov
|
||||
|
||||
bench-serving: ## Serving benchmark vs a live backend (BENCH_ARGS="--base-url ... --stream")
|
||||
$(PYTHON) scripts/bench_serving.py $(BENCH_ARGS)
|
||||
|
||||
lint: ## Run ruff linter and formatter check
|
||||
$(RUFF) check src tests
|
||||
$(RUFF) format --check src tests
|
||||
|
||||
Reference in New Issue
Block a user