From 99569e786ec8a9e1d6af63431a330762bf15f320 Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Fri, 7 Aug 2026 15:07:10 +0200 Subject: [PATCH] docs: correct stale tooling and model references Three migrations left their documentation behind: wakeup.sh was replaced by the Makefile during the project structure consolidation, but AGENTS.md and the e2e README still tell you to run it. The log path moved to build/logs/server.log at the same time. The local model moved to gemma4:e2b, but the e2e prerequisites and the benchmark recommendation still name mistral-nemo. The benchmark figures in CLAUDE.md predate the current model. Measured 2026-08-07: ~95 tok/s, full flow ~10-13s for simple turns, cold model load ~36s rather than ~8s. A turn costs three sequential Ollama calls and ~710 generated tokens regardless of how trivial the question is. Co-Authored-By: Claude --- AGENTS.md | 4 ++-- CLAUDE.md | 2 +- scripts/benchmark_steward.py | 2 +- tests/e2e/README.md | 10 +++++----- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 05ba29c..bfe9f79 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,8 +17,8 @@ This document contains instructions and documentation references for AI assistan ### ๐Ÿงช Local Development Setup * **Always test locally first** before committing and deploying. The build-deploy loop is slow. -* **Start the local server** with `./wakeup.sh` - logs are written to `logs/server.log` for easy tailing -* **Auto-reload**: The wakeup script runs uvicorn in reload mode - code changes are picked up automatically without restart (except for requirements.txt changes) +* **Start the local server** with `make run` - logs are written to `build/logs/server.log` for easy tailing +* **Auto-reload**: `make run` runs uvicorn in reload mode - code changes are picked up automatically without restart (except for dependency changes) * **Test REST endpoints** against `http://localhost:8777` using curl or similar tools * **Only deploy** when a phase or feature is complete and tested locally * **Environment**: Copy `.env.example` to `.env` and configure for your local setup (Ollama, Redis, Qdrant hosts) diff --git a/CLAUDE.md b/CLAUDE.md index 2c8abe2..76a9af8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ Dependencies are in `pyproject.toml` (`[project.dependencies]` and `[project.opt **Claude Sonnet 5+ rejects sampling parameters.** `temperature`/`top_p`/`top_k` return a 400. Use `get_sampling_settings()` from the model selector instead of passing `ModelSettings(temperature=...)` directly to agents that can run on the Claude fallback. The contract test suite pins this (`make test-contracts`). -**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). Current GPU-resident numbers (2026-07-14, driver 570, gemma4:e2b at ~100 tok/s): Steward analysis ~6s warm, full Steward โ†’ orchestrate โ†’ synthesize flow 11โ€“25s, librarian-routed queries ~20-25s. The old "~35s steward / ~2 min flow" figures were measured during the CPU-only era (driver mismatch, 13 tok/s) โ€” do not plan against them. Cold start after 2h idle adds ~8s (`OLLAMA_KEEP_ALIVE=2h`). `STEWARD_TIMEOUT` defaults to 60s. +**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). Current GPU-resident numbers (measured 2026-08-07, gemma4:e2b at ~95 tok/s): full Steward โ†’ orchestrate โ†’ synthesize flow ~10โ€“13s for simple turns; librarian-routed queries ~20-25s (not re-measured). A single turn costs **3 sequential Ollama calls and ~710 generated tokens** even for "what is 61 plus 12?" โ€” most of it the model's own reasoning, paid three times. Cold model load is ~36s, avoided while the model is pinned with `keep_alive: -1`; the `OLLAMA_KEEP_ALIVE=2h` default otherwise reintroduces it. The old "~35s steward / ~2 min flow" and "11โ€“25s flow" figures are superseded โ€” do not plan against them. `STEWARD_TIMEOUT` defaults to 60s. **`get_benchmark_store` does not exist.** The benchmarking module (`src/core/benchmarks.py`) was never implemented. `scripts/benchmark_analysis.py` also references it and is broken. Do not add mocks for it in tests. diff --git a/scripts/benchmark_steward.py b/scripts/benchmark_steward.py index c61710f..863e68a 100755 --- a/scripts/benchmark_steward.py +++ b/scripts/benchmark_steward.py @@ -268,7 +268,7 @@ async def run_benchmarks(iterations: int = 10, verbose: bool = False): print(f" Max: {overall_max:.3f}s (target: โ‰ค5.0s)") print(f" Avg: {overall_avg:.3f}s (target: โ‰ค1.67s)") print(f"\n Recommendations:") - print(f" - Switch to a faster model (current: mistral-nemo)") + print(f" - Switch to a faster model (current: gemma4:e2b)") print(f" - Reduce system prompt complexity") print(f" - Limit tool calls (currently limited to 3)") print(f" - Consider caching household registry responses") diff --git a/tests/e2e/README.md b/tests/e2e/README.md index 866d1de..410df68 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -4,8 +4,8 @@ These tests make real HTTP requests to the running Tatlock API server to verify ## Prerequisites -1. **Server must be running** on `http://localhost:8777` (use `./wakeup.sh`) -2. **Ollama must be running** with `mistral-nemo:latest` model +1. **Server must be running** on `http://localhost:8777` (use `make run`) +2. **Ollama must be running** with the `gemma4:e2b` model 3. **Redis must be running** (for benchmarking) 4. **Qdrant must be running** on `http://localhost:6333` (for memory tests) @@ -15,9 +15,9 @@ These tests make real HTTP requests to the running Tatlock API server to verify ```bash # Terminal 1: Start the server (auto-reload enabled) -./wakeup.sh +make run -# Logs are written to logs/server.log - tail them in another terminal: +# Logs are written to build/logs/server.log - tail them in another terminal: tail -f logs/server.log ``` @@ -132,7 +132,7 @@ memory = await qdrant.find_memory_by_key("memories_llm_tester", "favorite_color" Make sure the server is running: ```bash -./wakeup.sh +make run curl http://localhost:8777/health # Should return 200 ```