From d2aeb8957bf81073c641314e723f378d9f8de3a6 Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Mon, 13 Jul 2026 18:35:53 +0200 Subject: [PATCH] docs: document local-first backend, gemma4 gotchas, and contract tests Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 19 +++++++++++++++++++ CLAUDE.md | 9 ++++++--- README.md | 22 +++++++++++++++------- docs/claude-integration.md | 2 +- docs/roadmap.md | 2 +- 5 files changed, 42 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b50cfa..c884195 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- **Local-first backend (claudification rollback)** - Ollama/gemma4 is now the primary backend; Claude remains as fallback. `PREFER_CLOUD_BACKEND` defaults to `false`, Claude is used automatically when the Ollama startup health check fails, and the Steward retries mid-request failures on the other backend in both directions +- **Default Claude model `claude-sonnet-5`** - `claude-sonnet-4-20250514` was retired by Anthropic on 2026-06-15 and would 404, leaving the fallback dead +- **Dedicated orchestration prompt** - `orchestrate_tool_calls()` now uses a terse tool-execution prompt (`TATLOCK_ORCHESTRATION_PROMPT`); the butler persona prompt suppressed gemma4 tool calling (the model reasoned about the calculator, then answered from memory with wrong arithmetic). Synthesis keeps the persona prompt, so user-visible voice is unchanged + +### Fixed + +- **Startup crash with broken anthropic package** - Anthropic SDK imports in the model selector are now lazy, so an incompatible `anthropic` install degrades to Ollama-only operation instead of crashing the app at import time (root cause of the production outage since April) +- **Claude Sonnet 5 rejects sampling parameters** - removed `temperature` from the Steward's direct Claude call and made the Housekeeper's temperature setting backend-conditional via `get_sampling_settings()` +- **Pin `anthropic>=0.77,<1.0`** - the April image resolved an anthropic version incompatible with pydantic-ai 1.27 +- **Steward timeout configurable** - new `STEWARD_TIMEOUT` (default 60s) replaces the hardcoded 30s, which gemma4 chronically exceeded (~35s warm analysis), causing every request to fail or fall back + +### Added + +- **Ollama startup health check** - verifies the server is reachable and `OLLAMA_DEFAULT_MODEL` is pulled; feeds backend resolution and `get_model_info()` +- **Contract tests** (`tests/contracts/`, `make test-contracts`) - wire-level tests that send the raw requests the code sends to Ollama (native + OpenAI-compat tool calling), Anthropic (including the pinned temperature-rejection contract), Qdrant, SearXNG, library-desk, and Redis; unreachable services skip, wrong response shapes fail +- **Backend resolution unit tests** (`tests/anthropic/`) + ## [2.2.0] - 2026-04-04 ### Changed diff --git a/CLAUDE.md b/CLAUDE.md index c348a9c..0a7babd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,6 +8,7 @@ Claude Code-specific notes for this project. For general development instruction make setup # Create venv and install all dependencies make test # Unit tests (no external services) make test-integration # Integration tests (needs Claude/Ollama) +make test-contracts # Wire-level contract tests against live service boundaries make run # Start dev server on port 8777 make lint # Ruff linter + formatter check make typecheck # Mypy @@ -18,13 +19,15 @@ Dependencies are in `pyproject.toml` (`[project.dependencies]` and `[project.opt ## Critical Gotchas -**ASGITransport does NOT trigger FastAPI lifespan events.** The session-scoped `_initialize_app` fixture in `tests/conftest.py` calls `initialize_application()` explicitly via `asyncio.run()`. Without this, `check_claude_health()` never runs and `_claude_available` stays `None`, causing all tests to silently fall back to Ollama. +**ASGITransport does NOT trigger FastAPI lifespan events.** The session-scoped `_initialize_app` fixture in `tests/conftest.py` calls `initialize_application()` explicitly via `asyncio.run()`. Without this, the Ollama/Claude health checks never run: `_ollama_available` stays `None` (treated as available, so requests go to Ollama) and `_claude_available` stays `None` (treated as unavailable, so the Claude fallback never engages). **AsyncIO scope mismatch.** `asyncio_default_fixture_loop_scope = function` is set in `pyproject.toml`. Session-scoped async fixtures cause `ScopeMismatch` errors. The fix is to use a sync fixture with `asyncio.run()` for session-scoped initialization. -**Ollama is unreliable for tool calling.** `mistral-nemo` on Ollama often does mental math instead of calling calculator tools, and frequently gets wrong answers. Claude reliably calls tools. If integration tests give wrong math answers, check which backend is actually being used. +**The butler persona prompt suppresses local-model tool calling.** With `TATLOCK_SYSTEM_PROMPT` attached, gemma4 reasons about calling the calculator, then answers from memory with wrong arithmetic (a different wrong product each run). `orchestrate_tool_calls()` therefore uses the terse `TATLOCK_ORCHESTRATION_PROMPT`; the persona is applied in `synthesize_from_results()`. Do not reattach the persona prompt to a tool-phase agent. `tool_choice: "required"` via extra_body does NOT force Ollama to call tools — it is advisory at best. -**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config. Ollama on tower-of-joy can be slow, especially on first request. +**Claude Sonnet 5+ rejects sampling parameters.** `temperature`/`top_p`/`top_k` return a 400. Use `get_sampling_settings()` from the model selector instead of passing `ModelSettings(temperature=...)` directly to agents that can run on the Claude fallback. The contract test suite pins this (`make test-contracts`). + +**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). The full local Steward → orchestrate → synthesize flow takes ~2 minutes on gemma4. Steward analysis alone needs ~35s warm — `STEWARD_TIMEOUT` defaults to 60s. **`get_benchmark_store` does not exist.** The benchmarking module (`src/core/benchmarks.py`) was never implemented. `scripts/benchmark_analysis.py` also references it and is broken. Do not add mocks for it in tests. diff --git a/README.md b/README.md index 9450c8a..b4bd78c 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ A privacy-first, offline-capable personal assistant system that coordinates spec - Error triggers for testing (rate_limit, context_overflow) - **Tatlock**: Real PydanticAI agent with butler personality - - **LLM Backend**: Ollama (mistral-nemo:latest by default) + - **LLM Backend**: Ollama (gemma4:e2b by default, local-first) with optional Claude fallback - **Personality**: Witty British butler, research-oriented - **Core Tools**: - **Calculator**: Safe mathematical expression evaluation @@ -74,7 +74,7 @@ A privacy-first, offline-capable personal assistant system that coordinates spec - Python 3.12+ (Python 3.12.11 recommended) - **External Services** (must be running separately): - - **Ollama**: LLM inference (mistral-nemo:latest, nomic-embed-text) + - **Ollama**: LLM inference (gemma4:e2b, nomic-embed-text) - **Redis**: Caching and session memory - **Qdrant**: Vector storage for The Biographer's memory - **SearXNG**: Web search (optional) @@ -264,7 +264,10 @@ Interactive documentation available at: pytest # Run unit tests only (no external services needed) -pytest --ignore=tests/e2e --ignore=tests/integration +pytest --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts + +# Wire-level contract tests against live service boundaries +make test-contracts # Run with coverage pytest --cov=src --cov-report=term-missing @@ -303,12 +306,17 @@ Create a `.env` file for custom configuration: API_HOST=0.0.0.0 API_PORT=8000 -# Ollama Configuration +# Ollama Configuration (primary backend) OLLAMA_HOST=http://localhost:11434 -OLLAMA_DEFAULT_MODEL=mistral-nemo:latest +OLLAMA_DEFAULT_MODEL=gemma4:e2b OLLAMA_EMBEDDING_MODEL=nomic-embed-text OLLAMA_TIMEOUT=120 +# Claude fallback (optional; used when Ollama is down or PREFER_CLOUD_BACKEND=true) +# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here +ANTHROPIC_MODEL=claude-sonnet-5 +PREFER_CLOUD_BACKEND=false + # Redis Configuration REDIS_HOST=localhost REDIS_PORT=6379 @@ -427,8 +435,8 @@ For LLM agent development guidelines and architectural decisions, see [AGENTS.md ## Version -Current version: **1.3.2** - Biographer tool type hints fix +Current version: see [CHANGELOG.md](CHANGELOG.md) --- -**Note**: Tatlock is a production-ready homelab butler. All household staff use PydanticAI with Ollama for local LLM inference. +**Note**: Tatlock is a production-ready homelab butler. All household staff use PydanticAI with local Ollama inference (gemma4), with an optional Claude cloud fallback. diff --git a/docs/claude-integration.md b/docs/claude-integration.md index f89ca14..b959c2d 100644 --- a/docs/claude-integration.md +++ b/docs/claude-integration.md @@ -3,7 +3,7 @@ ## Overview Tatlock uses a bidirectional Claude architecture: -- **Scenario A**: Tatlock powered by Claude backend (with Ollama fallback) — **COMPLETE** +- **Scenario A**: Tatlock powered by Claude backend (with Ollama fallback) — **COMPLETE**, then **rolled back to local-first**: Ollama/gemma4 is primary, Claude is retained as fallback (`PREFER_CLOUD_BACKEND=false`) - **Scenario B**: Tatlock exposed as MCP server for external Claude instances — **OPEN** - **Scenario C**: Offline operation via Ollama — **COMPLETE** diff --git a/docs/roadmap.md b/docs/roadmap.md index 831209c..d31e513 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -12,7 +12,7 @@ This document tracks open/planned work. Completed phases have been removed. - Household staff: Tatlock (Butler), Steward, Librarian, Biographer - Core tools: Calculator, Date/Time, Web search (SearXNG) - Memory system: Qdrant (vector), Redis (session cache), multi-tenancy via ContextVar -- Dual backend: Claude (preferred) + Ollama (fallback) +- Dual backend: Ollama/gemma4 (primary) + Claude (fallback) - 439 tests with good coverage ---