docs: correct stale tooling and model references
Three migrations left their documentation behind: wakeup.sh was replaced by the Makefile during the project structure consolidation, but AGENTS.md and the e2e README still tell you to run it. The log path moved to build/logs/server.log at the same time. The local model moved to gemma4:e2b, but the e2e prerequisites and the benchmark recommendation still name mistral-nemo. The benchmark figures in CLAUDE.md predate the current model. Measured 2026-08-07: ~95 tok/s, full flow ~10-13s for simple turns, cold model load ~36s rather than ~8s. A turn costs three sequential Ollama calls and ~710 generated tokens regardless of how trivial the question is. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -17,8 +17,8 @@ This document contains instructions and documentation references for AI assistan
|
||||
|
||||
### 🧪 Local Development Setup
|
||||
* **Always test locally first** before committing and deploying. The build-deploy loop is slow.
|
||||
* **Start the local server** with `./wakeup.sh` - logs are written to `logs/server.log` for easy tailing
|
||||
* **Auto-reload**: The wakeup script runs uvicorn in reload mode - code changes are picked up automatically without restart (except for requirements.txt changes)
|
||||
* **Start the local server** with `make run` - logs are written to `build/logs/server.log` for easy tailing
|
||||
* **Auto-reload**: `make run` runs uvicorn in reload mode - code changes are picked up automatically without restart (except for dependency changes)
|
||||
* **Test REST endpoints** against `http://localhost:8777` using curl or similar tools
|
||||
* **Only deploy** when a phase or feature is complete and tested locally
|
||||
* **Environment**: Copy `.env.example` to `.env` and configure for your local setup (Ollama, Redis, Qdrant hosts)
|
||||
|
||||
@@ -27,7 +27,7 @@ Dependencies are in `pyproject.toml` (`[project.dependencies]` and `[project.opt
|
||||
|
||||
**Claude Sonnet 5+ rejects sampling parameters.** `temperature`/`top_p`/`top_k` return a 400. Use `get_sampling_settings()` from the model selector instead of passing `ModelSettings(temperature=...)` directly to agents that can run on the Claude fallback. The contract test suite pins this (`make test-contracts`).
|
||||
|
||||
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). Current GPU-resident numbers (2026-07-14, driver 570, gemma4:e2b at ~100 tok/s): Steward analysis ~6s warm, full Steward → orchestrate → synthesize flow 11–25s, librarian-routed queries ~20-25s. The old "~35s steward / ~2 min flow" figures were measured during the CPU-only era (driver mismatch, 13 tok/s) — do not plan against them. Cold start after 2h idle adds ~8s (`OLLAMA_KEEP_ALIVE=2h`). `STEWARD_TIMEOUT` defaults to 60s.
|
||||
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). Current GPU-resident numbers (measured 2026-08-07, gemma4:e2b at ~95 tok/s): full Steward → orchestrate → synthesize flow ~10–13s for simple turns; librarian-routed queries ~20-25s (not re-measured). A single turn costs **3 sequential Ollama calls and ~710 generated tokens** even for "what is 61 plus 12?" — most of it the model's own reasoning, paid three times. Cold model load is ~36s, avoided while the model is pinned with `keep_alive: -1`; the `OLLAMA_KEEP_ALIVE=2h` default otherwise reintroduces it. The old "~35s steward / ~2 min flow" and "11–25s flow" figures are superseded — do not plan against them. `STEWARD_TIMEOUT` defaults to 60s.
|
||||
|
||||
**`get_benchmark_store` does not exist.** The benchmarking module (`src/core/benchmarks.py`) was never implemented. `scripts/benchmark_analysis.py` also references it and is broken. Do not add mocks for it in tests.
|
||||
|
||||
|
||||
@@ -268,7 +268,7 @@ async def run_benchmarks(iterations: int = 10, verbose: bool = False):
|
||||
print(f" Max: {overall_max:.3f}s (target: ≤5.0s)")
|
||||
print(f" Avg: {overall_avg:.3f}s (target: ≤1.67s)")
|
||||
print(f"\n Recommendations:")
|
||||
print(f" - Switch to a faster model (current: mistral-nemo)")
|
||||
print(f" - Switch to a faster model (current: gemma4:e2b)")
|
||||
print(f" - Reduce system prompt complexity")
|
||||
print(f" - Limit tool calls (currently limited to 3)")
|
||||
print(f" - Consider caching household registry responses")
|
||||
|
||||
+5
-5
@@ -4,8 +4,8 @@ These tests make real HTTP requests to the running Tatlock API server to verify
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Server must be running** on `http://localhost:8777` (use `./wakeup.sh`)
|
||||
2. **Ollama must be running** with `mistral-nemo:latest` model
|
||||
1. **Server must be running** on `http://localhost:8777` (use `make run`)
|
||||
2. **Ollama must be running** with the `gemma4:e2b` model
|
||||
3. **Redis must be running** (for benchmarking)
|
||||
4. **Qdrant must be running** on `http://localhost:6333` (for memory tests)
|
||||
|
||||
@@ -15,9 +15,9 @@ These tests make real HTTP requests to the running Tatlock API server to verify
|
||||
|
||||
```bash
|
||||
# Terminal 1: Start the server (auto-reload enabled)
|
||||
./wakeup.sh
|
||||
make run
|
||||
|
||||
# Logs are written to logs/server.log - tail them in another terminal:
|
||||
# Logs are written to build/logs/server.log - tail them in another terminal:
|
||||
tail -f logs/server.log
|
||||
```
|
||||
|
||||
@@ -132,7 +132,7 @@ memory = await qdrant.find_memory_by_key("memories_llm_tester", "favorite_color"
|
||||
|
||||
Make sure the server is running:
|
||||
```bash
|
||||
./wakeup.sh
|
||||
make run
|
||||
curl http://localhost:8777/health # Should return 200
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user