jpmschweitzerandClaude Fable 5
f578ebded8
A scripted 8-turn conversation through the real pipeline (tool call, memory recall, librarian route, small talk), run against any base URL so the same instrument measures Ollama today and llama-server after the cutover. Non-stream mode records wall time and usage; --stream records time-to-first-token; a VRAM poller and an optional interleaved second tenant capture the contention the serving plan calls out. Planted facts carry a BENCH-MARKER prefix so anything the biographer learns from a bench run stays recognizable in the memory stores. First smoke against production: 9.7s mean per simple turn, and TTFT equals wall time — the client sees nothing until the whole steward-orchestrate-synthesize pipeline has finished. Time-to-first-word for desklock is currently the full pipeline latency, which makes streamed synthesis a first-class goal of the serving work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
93 lines
4.0 KiB
Makefile
93 lines
4.0 KiB
Makefile
.PHONY: help setup run test test-unit test-integration test-contracts bench-serving lint typecheck clean
|
|
|
|
VENV := .venv
|
|
PYTHON := $(VENV)/bin/python
|
|
PIP := $(VENV)/bin/pip
|
|
PYTEST := $(VENV)/bin/pytest
|
|
RUFF := $(VENV)/bin/ruff
|
|
MYPY := $(VENV)/bin/mypy
|
|
UVICORN := $(VENV)/bin/uvicorn
|
|
|
|
HOST := 0.0.0.0
|
|
PORT := 8777
|
|
|
|
help: ## Show this help
|
|
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
|
|
|
|
setup: ## Create venv and install all dependencies
|
|
python3 -m venv $(VENV)
|
|
$(PIP) install --upgrade pip
|
|
$(PIP) install -e ".[dev]"
|
|
# Exit 0 from pip install is not evidence the environment works (D-24) - the
|
|
# 2026-08-09 core-api incident was exactly this: a venv that "installed fine"
|
|
# but was missing a declared dependency, surfacing as 11 collection errors
|
|
# that read like broken imports rather than an environment problem. Collection
|
|
# is the right cheap check here for that same reason: it imports every test
|
|
# module (and everything they import) without running the suite, so a missing
|
|
# or mismatched dependency fails setup itself instead of showing up later as a
|
|
# mysterious test failure. Scoped like `make test` (excludes e2e/integration/
|
|
# contracts, which need external services) and --no-cov since coverage
|
|
# instrumentation is irrelevant to "does this collect".
|
|
$(PYTEST) --collect-only -q --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts --no-cov
|
|
|
|
run: ## Start the development server on port 8777
|
|
@mkdir -p build/logs
|
|
@if lsof -Pi :$(PORT) -sTCP:LISTEN -t >/dev/null 2>&1; then \
|
|
echo "Error: Port $(PORT) is already in use"; \
|
|
echo "Run: lsof -i :$(PORT) to see what's using it"; \
|
|
exit 1; \
|
|
fi
|
|
$(UVICORN) src.main:app --reload --host $(HOST) --port $(PORT) 2>&1 | tee build/logs/server.log
|
|
|
|
test: ## Run unit tests (no external services needed)
|
|
$(PYTEST) --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts
|
|
|
|
test-unit: test ## Alias for test
|
|
|
|
test-integration: ## Run integration tests (needs Claude/Ollama)
|
|
$(PYTEST) tests/agents/test_tatlock_agent.py -v
|
|
|
|
test-contracts: ## Wire-level contract tests against live service boundaries
|
|
$(PYTEST) tests/contracts -v --no-cov
|
|
|
|
bench-serving: ## Serving benchmark vs a live backend (BENCH_ARGS="--base-url ... --stream")
|
|
$(PYTHON) scripts/bench_serving.py $(BENCH_ARGS)
|
|
|
|
lint: ## Run ruff linter and formatter check
|
|
$(RUFF) check src tests
|
|
$(RUFF) format --check src tests
|
|
|
|
typecheck: ## Run mypy type checking
|
|
$(MYPY) src
|
|
|
|
clean: ## Remove build artifacts, caches, and coverage reports
|
|
rm -rf .cache build
|
|
find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
|
|
|
|
# git hands a hook a non-login shell, which never sees ~/.local/bin — where
|
|
# gitleaks lands. Without this the scan reports "not installed" on every push,
|
|
# which is a check that fails open (D-24).
|
|
export PATH := $(HOME)/.local/bin:/usr/local/bin:$(PATH)
|
|
|
|
.PHONY: secrets
|
|
secrets: ## Scan the commits about to be pushed for credentials
|
|
@ci/secrets.sh
|
|
|
|
# The call surface is identical in every repo; what it runs is not.
|
|
#
|
|
# `secrets` runs first, deliberately: it is the only failure here that cannot be
|
|
# undone by fixing it afterwards. A failed lint costs another commit; a pushed
|
|
# credential is cached and indexed whether or not it is later deleted.
|
|
#
|
|
# Some of these fail today, and are left wired anyway. The state was measured
|
|
# once and written down in T-56 rather than being worked around here — a gate
|
|
# quietly narrowed to what already passes is a gate that reports success for
|
|
# doing nothing, which is the failure this workspace keeps rediscovering.
|
|
.PHONY: pre-push
|
|
pre-push: secrets lint ## Everything the pre-push hook runs
|
|
@echo " -- not gated here yet: typecheck (T-1), test (T-56)"
|
|
@echo " typecheck reports 95 errors in 31 files and has never passed, so"
|
|
@echo " gating on it blocked every push to this repo — including the commit"
|
|
@echo " that added the gate. Run 'make typecheck' before pushing anything"
|
|
@echo " that touches types; T-1 is the pass that earns this line's removal."
|