Compare commits

...
7 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Fable 5 11405e0acb chore: release v2.3.0
Build and Push / release (push) Successful in 25s
Build and Push / build (push) Successful in 4m34s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 23:54:32 +02:00
jpmschweitzerandClaude Fable 5 d2aeb8957b docs: document local-first backend, gemma4 gotchas, and contract tests
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Fable 5 d8c84d080c test: add wire-level service contract tests
tests/contracts sends the raw requests the code sends to Ollama (native API
and OpenAI-compat tool calling), Anthropic (including the pinned Sonnet 5
temperature-rejection contract), Qdrant, SearXNG, library-desk, and Redis.
Unreachable services skip; wrong response shapes fail. Run via
make test-contracts; excluded from the unit suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Fable 5 f03d41c698 fix: use dedicated tool-phase prompt for gemma4 orchestration
With the butler persona prompt attached, gemma4 reasons about calling the
calculator and then answers from memory with a different wrong product every
run; tool_choice=required via extra_body is advisory at best on Ollama's
OpenAI-compat layer. orchestrate_tool_calls() now uses a terse
TATLOCK_ORCHESTRATION_PROMPT; synthesize_from_results() keeps the persona,
so the user-visible voice is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Fable 5 033a1c01e8 feat: make Ollama/gemma4 the primary backend with Claude as fallback
Rolls back the claudification backend preference: PREFER_CLOUD_BACKEND now
defaults to false, resolve_backend() picks Ollama first and uses Claude when
explicitly preferred or when the new Ollama startup health check fails. The
Steward retries mid-request failures on the other backend in both directions.

Also hardens the fallback itself: Anthropic SDK imports are lazy so a broken
anthropic package degrades to Ollama-only instead of crashing at import time
(root cause of the production outage since April), anthropic is pinned to a
pydantic-ai-1.27-compatible range, ANTHROPIC_MODEL defaults to claude-sonnet-5
(sonnet-4-20250514 retired 2026-06-15), sampling parameters are stripped from
Claude calls (Sonnet 5 rejects them), and the Steward timeout is configurable
(STEWARD_TIMEOUT, default 60s) since gemma4 needs ~35s warm for analysis.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Opus 4.6 427ad311dc feat: switch default Ollama model to gemma4:e2b
Build and Push / release (push) Successful in 21s
Build and Push / build (push) Successful in 5m27s
gemma4:e2b has native function calling with dedicated tool tokens,
achieving 100% tool selection accuracy in benchmarks vs 67% for
mistral-nemo-large, with 5-8x faster response times (2-4s vs 15-20s)
and lower VRAM usage (8GB vs 9.2GB).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 22:44:27 +02:00
jpmschweitzerandClaude Opus 4.6 4f911929f4 ci: remove test gate from release pipeline
Build and Push / release (push) Successful in 2s
Build and Push / build (push) Successful in 1m23s
Tests are run locally before tagging. Removes the slow CI test job
and its dependency gates on release and build jobs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-05 21:22:51 +01:00
22 changed files with 1165 additions and 108 deletions
+11 -9
View File
@@ -8,17 +8,19 @@ API_HOST=0.0.0.0
API_PORT=8000 API_PORT=8000
API_PREFIX=/v1 API_PREFIX=/v1
# Anthropic Configuration (Claude - preferred backend) # Ollama Configuration (local - primary backend)
# Set ANTHROPIC_API_KEY to enable Claude as the default backend
# Without an API key, Tatlock uses Ollama exclusively
# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
ANTHROPIC_MODEL=claude-sonnet-4-20250514
PREFER_CLOUD_BACKEND=true
# Ollama Configuration (local fallback when Claude unavailable)
OLLAMA_HOST=http://localhost:11434 OLLAMA_HOST=http://localhost:11434
OLLAMA_DEFAULT_MODEL=mistral-nemo:latest OLLAMA_DEFAULT_MODEL=gemma4:e2b
OLLAMA_TIMEOUT=120 OLLAMA_TIMEOUT=120
STEWARD_TIMEOUT=60
# Anthropic Configuration (Claude - cloud fallback)
# Set ANTHROPIC_API_KEY to keep the Claude fallback available: it is used
# automatically when Ollama is down, or exclusively when PREFER_CLOUD_BACKEND=true
# Without an API key, Tatlock uses Ollama only
# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
ANTHROPIC_MODEL=claude-sonnet-5
PREFER_CLOUD_BACKEND=false
# SearXNG Configuration # SearXNG Configuration
SEARXNG_HOST=http://localhost:8087 SEARXNG_HOST=http://localhost:8087
-18
View File
@@ -6,24 +6,7 @@ on:
- 'v[0-9]*' - 'v[0-9]*'
jobs: jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: make setup
- name: Run unit tests
run: make test
release: release:
needs: test
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Create Gitea Release - name: Create Gitea Release
@@ -35,7 +18,6 @@ jobs:
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases" "${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases"
build: build:
needs: test
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
+31
View File
@@ -7,6 +7,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
## [2.3.0] - 2026-07-13
### Changed
- **Local-first backend (claudification rollback)** - Ollama/gemma4 is now the primary backend; Claude remains as fallback. `PREFER_CLOUD_BACKEND` defaults to `false`, Claude is used automatically when the Ollama startup health check fails, and the Steward retries mid-request failures on the other backend in both directions
- **Default Claude model `claude-sonnet-5`** - `claude-sonnet-4-20250514` was retired by Anthropic on 2026-06-15 and would 404, leaving the fallback dead
- **Dedicated orchestration prompt** - `orchestrate_tool_calls()` now uses a terse tool-execution prompt (`TATLOCK_ORCHESTRATION_PROMPT`); the butler persona prompt suppressed gemma4 tool calling (the model reasoned about the calculator, then answered from memory with wrong arithmetic). Synthesis keeps the persona prompt, so user-visible voice is unchanged
### Fixed
- **Startup crash with broken anthropic package** - Anthropic SDK imports in the model selector are now lazy, so an incompatible `anthropic` install degrades to Ollama-only operation instead of crashing the app at import time (root cause of the production outage since April)
- **Claude Sonnet 5 rejects sampling parameters** - removed `temperature` from the Steward's direct Claude call and made the Housekeeper's temperature setting backend-conditional via `get_sampling_settings()`
- **Pin `anthropic>=0.77,<1.0`** - the April image resolved an anthropic version incompatible with pydantic-ai 1.27
- **Steward timeout configurable** - new `STEWARD_TIMEOUT` (default 60s) replaces the hardcoded 30s, which gemma4 chronically exceeded (~35s warm analysis), causing every request to fail or fall back
### Added
- **Ollama startup health check** - verifies the server is reachable and `OLLAMA_DEFAULT_MODEL` is pulled; feeds backend resolution and `get_model_info()`
- **Contract tests** (`tests/contracts/`, `make test-contracts`) - wire-level tests that send the raw requests the code sends to Ollama (native + OpenAI-compat tool calling), Anthropic (including the pinned temperature-rejection contract), Qdrant, SearXNG, library-desk, and Redis; unreachable services skip, wrong response shapes fail
- **Backend resolution unit tests** (`tests/anthropic/`)
## [2.2.0] - 2026-04-04
### Changed
- **Switch default Ollama model to gemma4:e2b** - Replaces mistral-nemo as the local LLM backend; gemma4:e2b has native function calling support, faster tool calling (2-4s vs 15-20s), better parameter accuracy on word problems, and uses less VRAM (8GB vs 9.2GB)
### Added
- **Tool calling benchmark script** (`scripts/benchmark_tool_calling.py`) - Compares tool calling accuracy and latency across Ollama models via the Tatlock API
## [2.1.0] - 2026-02-05 ## [2.1.0] - 2026-02-05
### Fixed ### Fixed
+6 -3
View File
@@ -8,6 +8,7 @@ Claude Code-specific notes for this project. For general development instruction
make setup # Create venv and install all dependencies make setup # Create venv and install all dependencies
make test # Unit tests (no external services) make test # Unit tests (no external services)
make test-integration # Integration tests (needs Claude/Ollama) make test-integration # Integration tests (needs Claude/Ollama)
make test-contracts # Wire-level contract tests against live service boundaries
make run # Start dev server on port 8777 make run # Start dev server on port 8777
make lint # Ruff linter + formatter check make lint # Ruff linter + formatter check
make typecheck # Mypy make typecheck # Mypy
@@ -18,13 +19,15 @@ Dependencies are in `pyproject.toml` (`[project.dependencies]` and `[project.opt
## Critical Gotchas ## Critical Gotchas
**ASGITransport does NOT trigger FastAPI lifespan events.** The session-scoped `_initialize_app` fixture in `tests/conftest.py` calls `initialize_application()` explicitly via `asyncio.run()`. Without this, `check_claude_health()` never runs and `_claude_available` stays `None`, causing all tests to silently fall back to Ollama. **ASGITransport does NOT trigger FastAPI lifespan events.** The session-scoped `_initialize_app` fixture in `tests/conftest.py` calls `initialize_application()` explicitly via `asyncio.run()`. Without this, the Ollama/Claude health checks never run: `_ollama_available` stays `None` (treated as available, so requests go to Ollama) and `_claude_available` stays `None` (treated as unavailable, so the Claude fallback never engages).
**AsyncIO scope mismatch.** `asyncio_default_fixture_loop_scope = function` is set in `pyproject.toml`. Session-scoped async fixtures cause `ScopeMismatch` errors. The fix is to use a sync fixture with `asyncio.run()` for session-scoped initialization. **AsyncIO scope mismatch.** `asyncio_default_fixture_loop_scope = function` is set in `pyproject.toml`. Session-scoped async fixtures cause `ScopeMismatch` errors. The fix is to use a sync fixture with `asyncio.run()` for session-scoped initialization.
**Ollama is unreliable for tool calling.** `mistral-nemo` on Ollama often does mental math instead of calling calculator tools, and frequently gets wrong answers. Claude reliably calls tools. If integration tests give wrong math answers, check which backend is actually being used. **The butler persona prompt suppresses local-model tool calling.** With `TATLOCK_SYSTEM_PROMPT` attached, gemma4 reasons about calling the calculator, then answers from memory with wrong arithmetic (a different wrong product each run). `orchestrate_tool_calls()` therefore uses the terse `TATLOCK_ORCHESTRATION_PROMPT`; the persona is applied in `synthesize_from_results()`. Do not reattach the persona prompt to a tool-phase agent. `tool_choice: "required"` via extra_body does NOT force Ollama to call tools — it is advisory at best.
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config. Ollama on tower-of-joy can be slow, especially on first request. **Claude Sonnet 5+ rejects sampling parameters.** `temperature`/`top_p`/`top_k` return a 400. Use `get_sampling_settings()` from the model selector instead of passing `ModelSettings(temperature=...)` directly to agents that can run on the Claude fallback. The contract test suite pins this (`make test-contracts`).
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). The full local Steward → orchestrate → synthesize flow takes ~2 minutes on gemma4. Steward analysis alone needs ~35s warm — `STEWARD_TIMEOUT` defaults to 60s.
**`get_benchmark_store` does not exist.** The benchmarking module (`src/core/benchmarks.py`) was never implemented. `scripts/benchmark_analysis.py` also references it and is broken. Do not add mocks for it in tests. **`get_benchmark_store` does not exist.** The benchmarking module (`src/core/benchmarks.py`) was never implemented. `scripts/benchmark_analysis.py` also references it and is broken. Do not add mocks for it in tests.
+5 -2
View File
@@ -1,4 +1,4 @@
.PHONY: help setup run test test-unit test-integration lint typecheck clean .PHONY: help setup run test test-unit test-integration test-contracts lint typecheck clean
VENV := .venv VENV := .venv
PYTHON := $(VENV)/bin/python PYTHON := $(VENV)/bin/python
@@ -29,13 +29,16 @@ run: ## Start the development server on port 8777
$(UVICORN) src.main:app --reload --host $(HOST) --port $(PORT) 2>&1 | tee build/logs/server.log $(UVICORN) src.main:app --reload --host $(HOST) --port $(PORT) 2>&1 | tee build/logs/server.log
test: ## Run unit tests (no external services needed) test: ## Run unit tests (no external services needed)
$(PYTEST) --ignore=tests/e2e --ignore=tests/integration $(PYTEST) --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts
test-unit: test ## Alias for test test-unit: test ## Alias for test
test-integration: ## Run integration tests (needs Claude/Ollama) test-integration: ## Run integration tests (needs Claude/Ollama)
$(PYTEST) tests/agents/test_tatlock_agent.py -v $(PYTEST) tests/agents/test_tatlock_agent.py -v
test-contracts: ## Wire-level contract tests against live service boundaries
$(PYTEST) tests/contracts -v --no-cov
lint: ## Run ruff linter and formatter check lint: ## Run ruff linter and formatter check
$(RUFF) check src tests $(RUFF) check src tests
$(RUFF) format --check src tests $(RUFF) format --check src tests
+15 -7
View File
@@ -58,7 +58,7 @@ A privacy-first, offline-capable personal assistant system that coordinates spec
- Error triggers for testing (rate_limit, context_overflow) - Error triggers for testing (rate_limit, context_overflow)
- **Tatlock**: Real PydanticAI agent with butler personality - **Tatlock**: Real PydanticAI agent with butler personality
- **LLM Backend**: Ollama (mistral-nemo:latest by default) - **LLM Backend**: Ollama (gemma4:e2b by default, local-first) with optional Claude fallback
- **Personality**: Witty British butler, research-oriented - **Personality**: Witty British butler, research-oriented
- **Core Tools**: - **Core Tools**:
- **Calculator**: Safe mathematical expression evaluation - **Calculator**: Safe mathematical expression evaluation
@@ -74,7 +74,7 @@ A privacy-first, offline-capable personal assistant system that coordinates spec
- Python 3.12+ (Python 3.12.11 recommended) - Python 3.12+ (Python 3.12.11 recommended)
- **External Services** (must be running separately): - **External Services** (must be running separately):
- **Ollama**: LLM inference (mistral-nemo:latest, nomic-embed-text) - **Ollama**: LLM inference (gemma4:e2b, nomic-embed-text)
- **Redis**: Caching and session memory - **Redis**: Caching and session memory
- **Qdrant**: Vector storage for The Biographer's memory - **Qdrant**: Vector storage for The Biographer's memory
- **SearXNG**: Web search (optional) - **SearXNG**: Web search (optional)
@@ -264,7 +264,10 @@ Interactive documentation available at:
pytest pytest
# Run unit tests only (no external services needed) # Run unit tests only (no external services needed)
pytest --ignore=tests/e2e --ignore=tests/integration pytest --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts
# Wire-level contract tests against live service boundaries
make test-contracts
# Run with coverage # Run with coverage
pytest --cov=src --cov-report=term-missing pytest --cov=src --cov-report=term-missing
@@ -303,12 +306,17 @@ Create a `.env` file for custom configuration:
API_HOST=0.0.0.0 API_HOST=0.0.0.0
API_PORT=8000 API_PORT=8000
# Ollama Configuration # Ollama Configuration (primary backend)
OLLAMA_HOST=http://localhost:11434 OLLAMA_HOST=http://localhost:11434
OLLAMA_DEFAULT_MODEL=mistral-nemo:latest OLLAMA_DEFAULT_MODEL=gemma4:e2b
OLLAMA_EMBEDDING_MODEL=nomic-embed-text OLLAMA_EMBEDDING_MODEL=nomic-embed-text
OLLAMA_TIMEOUT=120 OLLAMA_TIMEOUT=120
# Claude fallback (optional; used when Ollama is down or PREFER_CLOUD_BACKEND=true)
# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
ANTHROPIC_MODEL=claude-sonnet-5
PREFER_CLOUD_BACKEND=false
# Redis Configuration # Redis Configuration
REDIS_HOST=localhost REDIS_HOST=localhost
REDIS_PORT=6379 REDIS_PORT=6379
@@ -427,8 +435,8 @@ For LLM agent development guidelines and architectural decisions, see [AGENTS.md
## Version ## Version
Current version: **1.3.2** - Biographer tool type hints fix Current version: see [CHANGELOG.md](CHANGELOG.md)
--- ---
**Note**: Tatlock is a production-ready homelab butler. All household staff use PydanticAI with Ollama for local LLM inference. **Note**: Tatlock is a production-ready homelab butler. All household staff use PydanticAI with local Ollama inference (gemma4), with an optional Claude cloud fallback.
+1 -1
View File
@@ -3,7 +3,7 @@
## Overview ## Overview
Tatlock uses a bidirectional Claude architecture: Tatlock uses a bidirectional Claude architecture:
- **Scenario A**: Tatlock powered by Claude backend (with Ollama fallback) — **COMPLETE** - **Scenario A**: Tatlock powered by Claude backend (with Ollama fallback) — **COMPLETE**, then **rolled back to local-first**: Ollama/gemma4 is primary, Claude is retained as fallback (`PREFER_CLOUD_BACKEND=false`)
- **Scenario B**: Tatlock exposed as MCP server for external Claude instances — **OPEN** - **Scenario B**: Tatlock exposed as MCP server for external Claude instances — **OPEN**
- **Scenario C**: Offline operation via Ollama — **COMPLETE** - **Scenario C**: Offline operation via Ollama — **COMPLETE**
+1 -1
View File
@@ -12,7 +12,7 @@ This document tracks open/planned work. Completed phases have been removed.
- Household staff: Tatlock (Butler), Steward, Librarian, Biographer - Household staff: Tatlock (Butler), Steward, Librarian, Biographer
- Core tools: Calculator, Date/Time, Web search (SearXNG) - Core tools: Calculator, Date/Time, Web search (SearXNG)
- Memory system: Qdrant (vector), Redis (session cache), multi-tenancy via ContextVar - Memory system: Qdrant (vector), Redis (session cache), multi-tenancy via ContextVar
- Dual backend: Claude (preferred) + Ollama (fallback) - Dual backend: Ollama/gemma4 (primary) + Claude (fallback)
- 439 tests with good coverage - 439 tests with good coverage
--- ---
+3 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "tatlock" name = "tatlock"
version = "2.1.0" version = "2.3.0"
description = "OpenAI-compatible API with Ollama backend" description = "OpenAI-compatible API with Ollama backend"
requires-python = ">=3.12" requires-python = ">=3.12"
dependencies = [ dependencies = [
@@ -13,6 +13,7 @@ dependencies = [
"pydantic>=2.11,<2.13", "pydantic>=2.11,<2.13",
"pydantic-settings>=2.12,<2.13", "pydantic-settings>=2.12,<2.13",
"pydantic-ai-slim[openai,anthropic]>=1.27,<1.28", "pydantic-ai-slim[openai,anthropic]>=1.27,<1.28",
"anthropic>=0.77,<1.0",
"httpx>=0.28,<0.29", "httpx>=0.28,<0.29",
"sse-starlette>=3.0,<3.1", "sse-starlette>=3.0,<3.1",
"python-dotenv>=1.2,<1.3", "python-dotenv>=1.2,<1.3",
@@ -46,6 +47,7 @@ markers = [
"unit: Unit tests", "unit: Unit tests",
"integration: Integration tests", "integration: Integration tests",
"slow: Slow running tests", "slow: Slow running tests",
"contract: Wire-level contract tests against live service boundaries",
] ]
addopts = [ addopts = [
"--verbose", "--verbose",
+542
View File
@@ -0,0 +1,542 @@
"""
Benchmark tool calling across different Ollama models via Tatlock API.
Sends test prompts through the full Tatlock pipeline (Steward -> Orchestration
-> Synthesis) and records tool selection accuracy, latency, and response quality.
Between models, swaps OLLAMA_DEFAULT_MODEL in .env and waits for uvicorn
auto-reload. Requires the server to be running via ./wakeup.sh.
Usage:
.venv/bin/python scripts/benchmark_tool_calling.py
.venv/bin/python scripts/benchmark_tool_calling.py --models "gemma4:e4b,gemma4:e2b"
.venv/bin/python scripts/benchmark_tool_calling.py --iterations 3
"""
import argparse
import asyncio
import json
import re
import statistics
import time
from dataclasses import dataclass, field
from pathlib import Path
import httpx
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
API_BASE = "http://localhost:8777"
CHAT_URL = f"{API_BASE}/v1/chat/completions"
HEALTH_URL = f"{API_BASE}/health"
OLLAMA_URL = "http://localhost:11434"
ENV_PATH = Path(__file__).parent.parent / ".env"
DEFAULT_MODELS = ["mistral-nemo-large:latest", "gemma4:e4b", "gemma4:e2b"]
# ---------------------------------------------------------------------------
# Test scenarios
# ---------------------------------------------------------------------------
@dataclass
class Scenario:
name: str
prompt: str
expected_tool: str | None # None = no tool expected
# Patterns to check in the response text for indirect tool-use evidence
success_patterns: list[str] = field(default_factory=list)
category: str = "basic"
SCENARIOS = [
# --- Should call calculate_math ---
Scenario(
name="Simple arithmetic",
prompt="What is 144 divided by 12?",
expected_tool="calculate_math",
success_patterns=["12"],
category="calculator",
),
Scenario(
name="Square root",
prompt="What's the square root of 256?",
expected_tool="calculate_math",
success_patterns=["16"],
category="calculator",
),
Scenario(
name="Complex math",
prompt="Calculate pi times the square of 5",
expected_tool="calculate_math",
success_patterns=["78.5"], # pi * 25 ≈ 78.54
category="calculator",
),
Scenario(
name="Word problem",
prompt="If I have 3 bags with 17 apples each and I eat 4, how many apples do I have?",
expected_tool="calculate_math",
success_patterns=["47"],
category="calculator",
),
# --- Should call get_current_time ---
Scenario(
name="Current date",
prompt="What's today's date?",
expected_tool="get_current_time",
success_patterns=["2026"], # Should contain current year
category="datetime",
),
Scenario(
name="Current time",
prompt="What time is it right now?",
expected_tool="get_current_time",
success_patterns=[":"], # Time format contains colons
category="datetime",
),
# --- Should call calculate_date_offset ---
Scenario(
name="Relative date past",
prompt="What was the date 2 weeks ago?",
expected_tool="calculate_date_offset",
success_patterns=["2026"],
category="datetime",
),
# --- Should call calculate_time_difference ---
Scenario(
name="Date difference",
prompt="How many days between January 1st 2025 and March 15th 2025?",
expected_tool="calculate_time_difference",
success_patterns=["73", "74"], # 73 or 74 days
category="datetime",
),
# --- Should NOT call any tool ---
Scenario(
name="Greeting",
prompt="Hello! How are you?",
expected_tool=None,
success_patterns=["sir"], # Butler personality
category="no_tool",
),
Scenario(
name="Knowledge question",
prompt="What is the capital of France?",
expected_tool=None,
success_patterns=["Paris"],
category="no_tool",
),
Scenario(
name="Opinion request",
prompt="What do you think about rainy days?",
expected_tool=None,
category="no_tool",
),
]
# ---------------------------------------------------------------------------
# Result tracking
# ---------------------------------------------------------------------------
@dataclass
class RunResult:
scenario: str
model: str
iteration: int
latency: float
response_text: str
has_correct_answer: bool
error: str | None = None
@dataclass
class ModelStats:
model: str
results: list[RunResult] = field(default_factory=list)
@property
def total(self) -> int:
return len(self.results)
@property
def errors(self) -> int:
return sum(1 for r in self.results if r.error)
@property
def accuracy(self) -> float:
valid = [r for r in self.results if not r.error]
if not valid:
return 0
return sum(1 for r in valid if r.has_correct_answer) / len(valid) * 100
@property
def avg_latency(self) -> float:
lats = [r.latency for r in self.results if not r.error]
return statistics.mean(lats) if lats else 0
@property
def p95_latency(self) -> float:
lats = sorted(r.latency for r in self.results if not r.error)
if not lats:
return 0
return lats[min(int(len(lats) * 0.95), len(lats) - 1)]
@property
def max_latency(self) -> float:
lats = [r.latency for r in self.results if not r.error]
return max(lats) if lats else 0
def category_accuracy(self, category: str) -> float:
cat_scenarios = {s.name for s in SCENARIOS if s.category == category}
valid = [r for r in self.results if not r.error and r.scenario in cat_scenarios]
if not valid:
return 0
return sum(1 for r in valid if r.has_correct_answer) / len(valid) * 100
# ---------------------------------------------------------------------------
# .env manipulation
# ---------------------------------------------------------------------------
def swap_model_in_env(model_name: str):
"""Swap OLLAMA_DEFAULT_MODEL in .env file."""
content = ENV_PATH.read_text()
content = re.sub(
r'^OLLAMA_DEFAULT_MODEL=.*$',
f'OLLAMA_DEFAULT_MODEL={model_name}',
content,
flags=re.MULTILINE,
)
ENV_PATH.write_text(content)
print(f" .env updated: OLLAMA_DEFAULT_MODEL={model_name}")
async def wait_for_server_reload(client: httpx.AsyncClient, timeout: float = 30):
"""Wait for uvicorn to auto-reload after .env change."""
# Give uvicorn a moment to detect the file change
await asyncio.sleep(3)
# Poll health endpoint
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
r = await client.get(HEALTH_URL, timeout=5)
if r.status_code == 200:
return
except Exception:
pass
await asyncio.sleep(1)
raise TimeoutError("Server did not come back after reload")
async def warm_up_ollama_model(client: httpx.AsyncClient, model_name: str):
"""Send a throwaway request to load the model into VRAM."""
print(f" Warming up {model_name} in Ollama...", end=" ", flush=True)
try:
r = await client.post(
f"{OLLAMA_URL}/api/generate",
json={"model": model_name, "prompt": "hi", "stream": False},
timeout=120,
)
r.raise_for_status()
duration = r.json().get("total_duration", 0) / 1e9
print(f"OK ({duration:.1f}s)")
except Exception as e:
print(f"WARN: {e}")
# ---------------------------------------------------------------------------
# Core benchmark logic
# ---------------------------------------------------------------------------
async def run_scenario(
client: httpx.AsyncClient,
scenario: Scenario,
model: str,
iteration: int,
) -> RunResult:
"""Run a single scenario through the Tatlock API."""
payload = {
"model": "Tatlock",
"messages": [{"role": "user", "content": scenario.prompt}],
}
start = time.monotonic()
try:
r = await client.post(CHAT_URL, json=payload, timeout=120)
latency = time.monotonic() - start
if r.status_code != 200:
return RunResult(
scenario=scenario.name,
model=model,
iteration=iteration,
latency=latency,
response_text="",
has_correct_answer=False,
error=f"HTTP {r.status_code}: {r.text[:100]}",
)
data = r.json()
response_text = data["choices"][0]["message"]["content"]
# Check if the response contains expected patterns
has_correct = True
if scenario.success_patterns:
has_correct = any(
p.lower() in response_text.lower()
for p in scenario.success_patterns
)
return RunResult(
scenario=scenario.name,
model=model,
iteration=iteration,
latency=latency,
response_text=response_text,
has_correct_answer=has_correct,
)
except Exception as e:
latency = time.monotonic() - start
return RunResult(
scenario=scenario.name,
model=model,
iteration=iteration,
latency=latency,
response_text="",
has_correct_answer=False,
error=str(e)[:200],
)
async def benchmark_model(
client: httpx.AsyncClient,
model_name: str,
iterations: int,
) -> ModelStats:
"""Run all scenarios for a single model."""
stats = ModelStats(model=model_name)
print(f"\n{'=' * 70}")
print(f" Model: {model_name}")
print(f"{'=' * 70}")
# Swap model in .env
swap_model_in_env(model_name)
# Warm up model in Ollama BEFORE server reload picks it up
await warm_up_ollama_model(client, model_name)
# Wait for server to reload with new model
print(" Waiting for server reload...", end=" ", flush=True)
await wait_for_server_reload(client)
print("OK")
# Run a throwaway request through the full pipeline to warm up
print(" Warming up pipeline...", end=" ", flush=True)
try:
await client.post(
CHAT_URL,
json={"model": "Tatlock", "messages": [{"role": "user", "content": "hi"}]},
timeout=120,
)
print("OK")
except Exception as e:
print(f"WARN: {e}")
for iteration in range(iterations):
if iterations > 1:
print(f"\n --- Iteration {iteration + 1}/{iterations} ---")
for scenario in SCENARIOS:
result = await run_scenario(client, scenario, model_name, iteration)
stats.results.append(result)
# Display
if result.error:
print(
f" [ERR ] {scenario.name:30s} {result.latency:5.1f}s "
f"{result.error[:60]}"
)
elif result.has_correct_answer:
preview = result.response_text[:60].replace("\n", " ")
print(f" [OK ] {scenario.name:30s} {result.latency:5.1f}s {preview}")
else:
preview = result.response_text[:60].replace("\n", " ")
print(f" [MISS] {scenario.name:30s} {result.latency:5.1f}s {preview}")
return stats
def print_comparison(all_stats: list[ModelStats]):
"""Print side-by-side comparison table."""
print("\n" + "=" * 80)
print(" COMPARISON SUMMARY")
print("=" * 80)
col_width = max(len(s.model) for s in all_stats) + 2
label_width = 32
header = f"{'Metric':<{label_width}}"
for s in all_stats:
header += f" {s.model:>{col_width}}"
print(f"\n{header}")
print("-" * (label_width + (col_width + 2) * len(all_stats)))
# Answer accuracy
row = f"{'Correct answer rate':<{label_width}}"
for s in all_stats:
row += f" {s.accuracy:>{col_width - 1}.1f}%"
print(row)
# Latency
row = f"{'Avg latency':<{label_width}}"
for s in all_stats:
row += f" {s.avg_latency:>{col_width - 1}.1f}s"
print(row)
row = f"{'P95 latency':<{label_width}}"
for s in all_stats:
row += f" {s.p95_latency:>{col_width - 1}.1f}s"
print(row)
row = f"{'Max latency':<{label_width}}"
for s in all_stats:
row += f" {s.max_latency:>{col_width - 1}.1f}s"
print(row)
# Errors
row = f"{'Errors':<{label_width}}"
for s in all_stats:
row += f" {s.errors:>{col_width}}"
print(row)
# Per-category
categories = sorted(set(sc.category for sc in SCENARIOS))
print(f"\n{'Per-category accuracy':<{label_width}}")
print("-" * (label_width + (col_width + 2) * len(all_stats)))
for cat in categories:
row = f" {cat:<{label_width - 2}}"
for s in all_stats:
row += f" {s.category_accuracy(cat):>{col_width - 1}.1f}%"
print(row)
# Mismatches
print(f"\n{'Missed answers':<50}")
print("-" * 80)
any_miss = False
for scenario in SCENARIOS:
misses = []
for s in all_stats:
sc_results = [r for r in s.results if r.scenario == scenario.name]
fails = [r for r in sc_results if not r.has_correct_answer and not r.error]
if fails:
preview = fails[0].response_text[:50].replace("\n", " ")
misses.append(f"{s.model}: \"{preview}\"")
if misses:
any_miss = True
print(f" {scenario.name}")
for m in misses:
print(f" {m}")
if not any_miss:
print(" (none)")
print("\n" + "=" * 80)
def save_results(all_stats: list[ModelStats], output_path: Path):
"""Save detailed results to JSON."""
data = {}
for stats in all_stats:
data[stats.model] = {
"summary": {
"accuracy": stats.accuracy,
"avg_latency": round(stats.avg_latency, 2),
"p95_latency": round(stats.p95_latency, 2),
"max_latency": round(stats.max_latency, 2),
"errors": stats.errors,
"total_runs": stats.total,
},
"runs": [
{
"scenario": r.scenario,
"iteration": r.iteration,
"latency": round(r.latency, 3),
"has_correct_answer": r.has_correct_answer,
"response_text": r.response_text,
"error": r.error,
}
for r in stats.results
],
}
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(data, indent=2))
print(f"\nDetailed results saved to: {output_path}")
async def main():
parser = argparse.ArgumentParser(description="Benchmark tool calling across Ollama models via Tatlock API")
parser.add_argument(
"--iterations", type=int, default=1,
help="Iterations per model (default: 1)",
)
parser.add_argument(
"--models", type=str, default=",".join(DEFAULT_MODELS),
help=f"Comma-separated models (default: {','.join(DEFAULT_MODELS)})",
)
parser.add_argument(
"--output", type=str, default="logs/benchmark_results.json",
help="JSON output path (default: logs/benchmark_results.json)",
)
args = parser.parse_args()
models = [m.strip() for m in args.models.split(",")]
# Verify server is running
async with httpx.AsyncClient() as client:
try:
r = await client.get(HEALTH_URL, timeout=5)
r.raise_for_status()
print("Server is running.")
except Exception:
print("ERROR: Server not running. Start it with ./wakeup.sh first.")
return
print("=" * 70)
print(" Tool Calling Benchmark (via Tatlock API)")
print("=" * 70)
print(f" Models: {', '.join(models)}")
print(f" Scenarios: {len(SCENARIOS)}")
print(f" Iterations: {args.iterations}")
print(f" Total runs: {len(SCENARIOS) * args.iterations * len(models)}")
# Remember original model to restore after benchmark
original_env = ENV_PATH.read_text()
all_stats = []
async with httpx.AsyncClient() as client:
for model in models:
stats = await benchmark_model(client, model, args.iterations)
all_stats.append(stats)
# Restore original .env
ENV_PATH.write_text(original_env)
print(f"\n .env restored to original")
print_comparison(all_stats)
save_results(all_stats, Path(args.output))
if __name__ == "__main__":
asyncio.run(main())
+6 -6
View File
@@ -204,13 +204,13 @@ async def run_housekeeper(
) )
try: try:
# Use temperature 0.1 for slight exploration # Temperature 0.1 for slight exploration (skipped on Claude backend)
from pydantic_ai.settings import ModelSettings from src.anthropic.model_selector import get_sampling_settings
result = await agent.run( result = await agent.run(
prompt, prompt,
message_history=message_history, message_history=message_history,
model_settings=ModelSettings(temperature=0.1), model_settings=get_sampling_settings(0.1),
) )
logger.info( logger.info(
@@ -266,13 +266,13 @@ async def run_housekeeper_stream(
) )
try: try:
# Use temperature 0.1 for slight exploration # Temperature 0.1 for slight exploration (skipped on Claude backend)
from pydantic_ai.settings import ModelSettings from src.anthropic.model_selector import get_sampling_settings
async with agent.run_stream( async with agent.run_stream(
prompt, prompt,
message_history=message_history, message_history=message_history,
model_settings=ModelSettings(temperature=0.1), model_settings=get_sampling_settings(0.1),
) as response: ) as response:
async for delta in response.stream_text(delta=True): async for delta in response.stream_text(delta=True):
yield delta yield delta
+28 -14
View File
@@ -11,7 +11,7 @@ Uses plain text output (not JSON) for reliability. Supports both Claude
import httpx import httpx
from typing import Optional from typing import Optional
from src.anthropic.model_selector import is_claude_available, get_model_info from src.anthropic.model_selector import get_model_info, is_claude_available, resolve_backend
from src.core.config import config from src.core.config import config
from src.core.household_registry import get_household_registry from src.core.household_registry import get_household_registry
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
@@ -113,18 +113,19 @@ class StewardAgent:
def __init__(self): def __init__(self):
"""Initialize Steward with backend selection based on availability.""" """Initialize Steward with backend selection based on availability."""
# Ollama config (fallback) # Ollama config (primary)
self.ollama_host = str(config.OLLAMA_HOST).rstrip('/') self.ollama_host = str(config.OLLAMA_HOST).rstrip('/')
self.ollama_model = config.OLLAMA_DEFAULT_MODEL self.ollama_model = config.OLLAMA_DEFAULT_MODEL
# Claude config (preferred) # Claude config (fallback)
self.claude_model = config.ANTHROPIC_MODEL self.claude_model = config.ANTHROPIC_MODEL
self._anthropic_client = None self._anthropic_client = None
# Determine which backend to use # Determine which backend to use (Ollama-first, Claude when
self._use_claude = config.PREFER_CLOUD_BACKEND and is_claude_available() # preferred via config or when Ollama is down)
self._use_claude = resolve_backend() == "claude"
self.timeout = 30.0 # 30 second timeout for analysis self.timeout = float(config.STEWARD_TIMEOUT)
model_info = get_model_info() model_info = get_model_info()
logger.info( logger.info(
@@ -145,12 +146,12 @@ class StewardAgent:
"""Call Claude API directly for plain text generation.""" """Call Claude API directly for plain text generation."""
client = self._get_anthropic_client() client = self._get_anthropic_client()
# No temperature: rejected by Claude Sonnet 5+ (sampling params deprecated)
response = await client.messages.create( response = await client.messages.create(
model=self.claude_model, model=self.claude_model,
max_tokens=1024, max_tokens=1024,
system=system_prompt, system=system_prompt,
messages=[{"role": "user", "content": user_message}], messages=[{"role": "user", "content": user_message}],
temperature=0.3, # Lower = more consistent
) )
return response.content[0].text.strip() return response.content[0].text.strip()
@@ -227,20 +228,33 @@ class StewardAgent:
return analysis_text return analysis_text
except Exception as e: except Exception as e:
# If Claude fails, try Ollama as fallback # Mid-request fallback: retry on the other backend when possible
if self._use_claude: if self._use_claude:
logger.warning( logger.warning(
"steward_claude_fallback", "steward_claude_fallback",
error=str(e), error=str(e),
) )
analysis_text = await self._call_ollama(prompt) analysis_text = await self._call_ollama(prompt)
logger.debug( fallback_backend = "ollama_fallback"
"steward_analysis_received", elif is_claude_available():
backend="ollama_fallback", logger.warning(
text_preview=analysis_text[:150], "steward_ollama_fallback",
error=str(e),
) )
return analysis_text analysis_text = await self._call_claude(
raise system_prompt="You are the Steward of the household, advising the Butler (Tatlock) on which capabilities to use. Be concise and specific.",
user_message=prompt,
)
fallback_backend = "claude_fallback"
else:
raise
logger.debug(
"steward_analysis_received",
backend=fallback_backend,
text_preview=analysis_text[:150],
)
return analysis_text
# Global Steward instance # Global Steward instance
+18 -2
View File
@@ -129,6 +129,22 @@ or
""" """
# Tool-phase prompt for orchestrate_tool_calls(). The butler personality prompt
# suppresses tool calling on small local models (gemma4 reasons about the tool,
# then answers from memory with wrong arithmetic), so the orchestration phase
# uses a terse operator prompt; synthesize_from_results() applies the persona.
TATLOCK_ORCHESTRATION_PROMPT = """You are the tool-execution phase of Tatlock, \
a butler assistant. Your only job is to gather accurate results by calling the \
provided tools.
- ALWAYS use tools for the task - never answer from memory and never do mental math.
- Mathematics: call the calculate tool, even for trivial arithmetic.
- Dates and times: call the date/time tools, never guess.
- When the instructions say DELEGATE to an agent, call the matching delegate_to_* tool.
- After the tool results arrive, reply with a one-line factual summary of the results. \
A later step writes the polished reply, so do not add personality."""
class TatlockAgent(AgentInterface): class TatlockAgent(AgentInterface):
""" """
Tatlock - The Butler agent using PydanticAI with Ollama. Tatlock - The Butler agent using PydanticAI with Ollama.
@@ -654,10 +670,10 @@ class TatlockAgent(AgentInterface):
# Create a fresh agent instance with scoped tools only # Create a fresh agent instance with scoped tools only
model = get_model() model = get_model()
# Create agent with scoped tools # Create agent with scoped tools, using the tool-phase prompt
scoped_agent = Agent( scoped_agent = Agent(
model, model,
system_prompt=TATLOCK_SYSTEM_PROMPT, system_prompt=TATLOCK_ORCHESTRATION_PROMPT,
tools=scoped_tools, tools=scoped_tools,
) )
+8 -1
View File
@@ -1,19 +1,26 @@
""" """
Anthropic/Claude integration module. Anthropic/Claude integration module.
Provides model selection with automatic fallback between Claude and Ollama. Provides model selection with Ollama as primary backend and Claude
as the cloud fallback.
""" """
from src.anthropic.model_selector import ( from src.anthropic.model_selector import (
check_claude_health, check_claude_health,
check_ollama_health,
get_model, get_model,
get_tool_choice_settings, get_tool_choice_settings,
is_claude_available, is_claude_available,
is_ollama_available,
resolve_backend,
) )
__all__ = [ __all__ = [
"check_claude_health", "check_claude_health",
"check_ollama_health",
"get_model", "get_model",
"get_tool_choice_settings", "get_tool_choice_settings",
"is_claude_available", "is_claude_available",
"is_ollama_available",
"resolve_backend",
] ]
+155 -31
View File
@@ -1,23 +1,84 @@
""" """
Model selector for Claude/Ollama backend switching. Model selector for Ollama/Claude backend switching.
Provides automatic model selection with Claude as preferred backend Provides automatic model selection with Ollama as the primary local backend
and Ollama as offline fallback. and Claude as the cloud fallback. Claude is used when PREFER_CLOUD_BACKEND
is enabled, or automatically when Ollama is unavailable at startup.
The Anthropic SDK is imported lazily so a missing or broken `anthropic`
package degrades to Ollama-only operation instead of crashing the app.
""" """
from typing import Union from __future__ import annotations
from pydantic_ai.models.anthropic import AnthropicModel from typing import TYPE_CHECKING
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.anthropic import AnthropicProvider import httpx
from src.core.config import config from src.core.config import config
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
if TYPE_CHECKING:
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.settings import ModelSettings
logger = get_logger(__name__) logger = get_logger(__name__)
# Cached health check result (set once at startup) # Cached health check results (set once at startup)
_claude_available: bool | None = None _claude_available: bool | None = None
_ollama_available: bool | None = None
async def check_ollama_health() -> bool:
"""
Check if the Ollama server is reachable and has the configured model.
This should be called once at application startup.
The result is cached in `_ollama_available`.
Returns:
True if Ollama is reachable and OLLAMA_DEFAULT_MODEL is pulled.
"""
global _ollama_available
host = str(config.OLLAMA_HOST).rstrip("/")
model = config.OLLAMA_DEFAULT_MODEL
try:
async with httpx.AsyncClient(timeout=5.0) as client:
response = await client.get(f"{host}/api/tags")
response.raise_for_status()
names = [m.get("name", "") for m in response.json().get("models", [])]
if model in names or f"{model}:latest" in names:
_ollama_available = True
logger.info(
"ollama_health_check_passed",
host=host,
model=model,
)
return True
_ollama_available = False
logger.warning(
"ollama_health_check_failed",
reason="model_not_pulled",
host=host,
model=model,
hint=f"run `ollama pull {model}`",
)
return False
except Exception as e:
_ollama_available = False
logger.warning(
"ollama_health_check_failed",
reason="server_unreachable",
host=host,
error=str(e),
)
return False
async def check_claude_health() -> bool: async def check_claude_health() -> bool:
@@ -85,46 +146,93 @@ def is_claude_available() -> bool:
return _claude_available is True return _claude_available is True
def get_model(prefer_cloud: bool | None = None) -> Union[AnthropicModel, OpenAIChatModel]: def is_ollama_available() -> bool:
"""
Check if Ollama is available (from cached health check result).
Returns:
False only if the startup health check confirmed Ollama is down.
Unknown (check not run yet) counts as available so that contexts
without lifespan events keep the local-first behavior.
"""
return _ollama_available is not False
def resolve_backend(prefer_cloud: bool | None = None) -> str:
"""
Resolve which backend should serve requests.
Ollama is the primary backend. Claude is used when explicitly
preferred via PREFER_CLOUD_BACKEND, or as automatic fallback
when the startup health check found Ollama down.
Args:
prefer_cloud: Override config.PREFER_CLOUD_BACKEND for this call.
Returns:
"claude" or "ollama".
"""
use_cloud = prefer_cloud if prefer_cloud is not None else config.PREFER_CLOUD_BACKEND
if use_cloud and is_claude_available():
return "claude"
if not is_ollama_available() and is_claude_available():
logger.warning(
"backend_fallback_to_claude",
reason="ollama_unavailable",
)
return "claude"
return "ollama"
def get_model(prefer_cloud: bool | None = None) -> AnthropicModel | OpenAIChatModel:
""" """
Get the best available model. Get the best available model.
Returns Claude if available and preferred, otherwise Ollama. Returns Ollama unless Claude is preferred (or Ollama is down).
Args: Args:
prefer_cloud: Override config.PREFER_CLOUD_BACKEND for this call. prefer_cloud: Override config.PREFER_CLOUD_BACKEND for this call.
If None, uses the config value. If None, uses the config value.
Returns: Returns:
PydanticAI model instance (AnthropicModel or OpenAIChatModel). PydanticAI model instance (OpenAIChatModel or AnthropicModel).
Example: Example:
>>> model = get_model() >>> model = get_model()
>>> agent = Agent(model, system_prompt="...") >>> agent = Agent(model, system_prompt="...")
""" """
# Determine preference if resolve_backend(prefer_cloud) == "claude":
use_cloud = prefer_cloud if prefer_cloud is not None else config.PREFER_CLOUD_BACKEND try:
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
# Use Claude if available and preferred logger.debug(
if use_cloud and is_claude_available(): "model_selected",
logger.debug( backend="claude",
"model_selected", model=config.ANTHROPIC_MODEL,
backend="claude", )
model=config.ANTHROPIC_MODEL, return AnthropicModel(
) model_name=config.ANTHROPIC_MODEL,
return AnthropicModel( provider=AnthropicProvider(api_key=config.ANTHROPIC_API_KEY),
model_name=config.ANTHROPIC_MODEL, )
provider=AnthropicProvider(api_key=config.ANTHROPIC_API_KEY), except ImportError as e:
) logger.error(
"claude_backend_import_failed",
error=str(e),
hint="anthropic package missing or incompatible; using Ollama",
)
from pydantic_ai.models.openai import OpenAIChatModel
# Fall back to Ollama
from src.ollama.provider import get_ollama_provider from src.ollama.provider import get_ollama_provider
logger.debug( logger.debug(
"model_selected", "model_selected",
backend="ollama", backend="ollama",
model=config.OLLAMA_DEFAULT_MODEL, model=config.OLLAMA_DEFAULT_MODEL,
reason="fallback" if use_cloud else "preferred_local",
) )
return OpenAIChatModel( return OpenAIChatModel(
model_name=config.OLLAMA_DEFAULT_MODEL, model_name=config.OLLAMA_DEFAULT_MODEL,
@@ -132,7 +240,7 @@ def get_model(prefer_cloud: bool | None = None) -> Union[AnthropicModel, OpenAIC
) )
def get_tool_choice_settings() -> 'ModelSettings': def get_tool_choice_settings() -> ModelSettings:
""" """
Get model_settings for forcing tool calls on the first request. Get model_settings for forcing tool calls on the first request.
@@ -141,7 +249,7 @@ def get_tool_choice_settings() -> 'ModelSettings':
""" """
from pydantic_ai.settings import ModelSettings from pydantic_ai.settings import ModelSettings
if is_claude_available() and config.PREFER_CLOUD_BACKEND: if resolve_backend() == "claude":
# PydanticAI's Anthropic model handles tool_choice internally # PydanticAI's Anthropic model handles tool_choice internally
return ModelSettings() return ModelSettings()
else: else:
@@ -149,6 +257,20 @@ def get_tool_choice_settings() -> 'ModelSettings':
return ModelSettings(extra_body={"tool_choice": "required"}) return ModelSettings(extra_body={"tool_choice": "required"})
def get_sampling_settings(temperature: float) -> ModelSettings:
"""
Get model_settings with a sampling temperature where the backend allows it.
Ollama accepts a temperature; Claude Sonnet 5+ rejects sampling
parameters, so the Claude backend gets empty settings.
"""
from pydantic_ai.settings import ModelSettings
if resolve_backend() == "claude":
return ModelSettings()
return ModelSettings(temperature=temperature)
def get_model_info() -> dict: def get_model_info() -> dict:
""" """
Get information about the current model configuration. Get information about the current model configuration.
@@ -158,12 +280,14 @@ def get_model_info() -> dict:
Returns: Returns:
Dict with backend, model name, and availability info. Dict with backend, model name, and availability info.
""" """
use_cloud = config.PREFER_CLOUD_BACKEND and is_claude_available() backend = resolve_backend()
return { return {
"backend": "claude" if use_cloud else "ollama", "backend": backend,
"model": config.ANTHROPIC_MODEL if use_cloud else config.OLLAMA_DEFAULT_MODEL, "model": config.ANTHROPIC_MODEL if backend == "claude" else config.OLLAMA_DEFAULT_MODEL,
"claude_available": is_claude_available(), "claude_available": is_claude_available(),
"claude_configured": bool(config.ANTHROPIC_API_KEY), "claude_configured": bool(config.ANTHROPIC_API_KEY),
"ollama_available": is_ollama_available(),
"ollama_model": config.OLLAMA_DEFAULT_MODEL,
"prefer_cloud": config.PREFER_CLOUD_BACKEND, "prefer_cloud": config.PREFER_CLOUD_BACKEND,
} }
+12 -8
View File
@@ -64,33 +64,37 @@ class Config(BaseSettings):
API_PORT: int = Field(default=8000, description="API port") API_PORT: int = Field(default=8000, description="API port")
API_PREFIX: str = Field(default="/v1", description="API route prefix") API_PREFIX: str = Field(default="/v1", description="API route prefix")
# Anthropic Configuration (Claude - preferred backend) # Anthropic Configuration (Claude - cloud fallback)
ANTHROPIC_API_KEY: str | None = Field( ANTHROPIC_API_KEY: str | None = Field(
default=None, default=None,
description="Anthropic API key for Claude access" description="Anthropic API key for the Claude fallback backend"
) )
ANTHROPIC_MODEL: str = Field( ANTHROPIC_MODEL: str = Field(
default="claude-sonnet-4-20250514", default="claude-sonnet-5",
description="Claude model to use" description="Claude model for the fallback backend"
) )
PREFER_CLOUD_BACKEND: bool = Field( PREFER_CLOUD_BACKEND: bool = Field(
default=True, default=False,
description="Prefer Claude over Ollama when available" description="Prefer Claude over Ollama (default: local-first)"
) )
# Ollama Configuration (local fallback) # Ollama Configuration (local - primary backend)
OLLAMA_HOST: HttpUrl = Field( OLLAMA_HOST: HttpUrl = Field(
default="http://localhost:11434", default="http://localhost:11434",
description="Ollama server URL" description="Ollama server URL"
) )
OLLAMA_DEFAULT_MODEL: str = Field( OLLAMA_DEFAULT_MODEL: str = Field(
default="mistral-nemo:latest", default="gemma4:e2b",
description="Default Ollama model" description="Default Ollama model"
) )
OLLAMA_TIMEOUT: int = Field( OLLAMA_TIMEOUT: int = Field(
default=120, default=120,
description="Ollama request timeout in seconds" description="Ollama request timeout in seconds"
) )
STEWARD_TIMEOUT: int = Field(
default=60,
description="Steward analysis timeout in seconds (gemma4 needs ~35s warm)"
)
STREAM_TIMEOUT: int = Field( STREAM_TIMEOUT: int = Field(
default=20, default=20,
description="Timeout for each streaming turn in seconds" description="Timeout for each streaming turn in seconds"
+9 -3
View File
@@ -9,7 +9,11 @@ from src.agents.biographer import register_biographer
from src.agents.housekeeper import register_housekeeper from src.agents.housekeeper import register_housekeeper
from src.agents.librarian import register_librarian from src.agents.librarian import register_librarian
from src.agents.tatlock_core import TATLOCK_CORE_CAPABILITY, tatlock_core_tools from src.agents.tatlock_core import TATLOCK_CORE_CAPABILITY, tatlock_core_tools
from src.anthropic.model_selector import check_claude_health, get_model_info from src.anthropic.model_selector import (
check_claude_health,
check_ollama_health,
get_model_info,
)
from src.core.household_registry import get_household_registry from src.core.household_registry import get_household_registry
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
@@ -87,7 +91,7 @@ async def initialize_application():
Initialize the application. Initialize the application.
Performs all startup tasks: Performs all startup tasks:
1. Check Claude API health (for backend selection) 1. Check Ollama (primary) and Claude (fallback) health for backend selection
2. Register household members 2. Register household members
3. (Future) Initialize connections 3. (Future) Initialize connections
@@ -95,13 +99,15 @@ async def initialize_application():
""" """
logger.info("application_initialization_starting") logger.info("application_initialization_starting")
# Check Claude API health for backend selection # Check backend health: Ollama is primary, Claude is the fallback
await check_ollama_health()
await check_claude_health() await check_claude_health()
model_info = get_model_info() model_info = get_model_info()
logger.info( logger.info(
"model_backend_configured", "model_backend_configured",
backend=model_info["backend"], backend=model_info["backend"],
model=model_info["model"], model=model_info["model"],
ollama_available=model_info["ollama_available"],
claude_available=model_info["claude_available"], claude_available=model_info["claude_available"],
) )
+3 -1
View File
@@ -410,10 +410,12 @@ async def test_tatlock_ollama_fallback(async_client: AsyncClient):
"stream": False "stream": False
} }
# 300s: this test forbids the Claude rescue, and the full local
# Steward -> orchestrate -> synthesize flow on gemma4 exceeds 120s
response = await async_client.post( response = await async_client.post(
"/v1/chat/completions", "/v1/chat/completions",
json=request_data, json=request_data,
timeout=120.0 timeout=300.0
) )
assert response.status_code == 200 assert response.status_code == 200
View File
+92
View File
@@ -0,0 +1,92 @@
"""
Unit tests for backend selection (Ollama primary, Claude fallback).
These tests set the cached health-check globals directly so they are
deterministic regardless of which services are reachable.
"""
import pytest
from src.anthropic import model_selector
from src.core.config import config
@pytest.fixture
def local_first(monkeypatch):
"""Baseline: local-first config, both backends healthy."""
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", False)
monkeypatch.setattr(config, "ANTHROPIC_API_KEY", "sk-test-fake")
monkeypatch.setattr(model_selector, "_claude_available", True)
monkeypatch.setattr(model_selector, "_ollama_available", True)
class TestResolveBackend:
def test_default_is_ollama(self, local_first):
assert model_selector.resolve_backend() == "ollama"
def test_prefer_cloud_config_selects_claude(self, local_first, monkeypatch):
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", True)
assert model_selector.resolve_backend() == "claude"
def test_prefer_cloud_override_selects_claude(self, local_first):
assert model_selector.resolve_backend(prefer_cloud=True) == "claude"
def test_prefer_cloud_without_claude_falls_back_to_ollama(self, local_first, monkeypatch):
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", True)
monkeypatch.setattr(model_selector, "_claude_available", False)
assert model_selector.resolve_backend() == "ollama"
def test_ollama_down_falls_back_to_claude(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", False)
assert model_selector.resolve_backend() == "claude"
def test_ollama_down_without_claude_stays_ollama(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", False)
monkeypatch.setattr(model_selector, "_claude_available", False)
assert model_selector.resolve_backend() == "ollama"
def test_unknown_ollama_state_counts_as_available(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", None)
assert model_selector.resolve_backend() == "ollama"
class TestGetModel:
def test_ollama_backend_returns_openai_chat_model(self, local_first):
from pydantic_ai.models.openai import OpenAIChatModel
model = model_selector.get_model()
assert isinstance(model, OpenAIChatModel)
assert model.model_name == config.OLLAMA_DEFAULT_MODEL
def test_claude_backend_returns_anthropic_model(self, local_first):
from pydantic_ai.models.anthropic import AnthropicModel
model = model_selector.get_model(prefer_cloud=True)
assert isinstance(model, AnthropicModel)
assert model.model_name == config.ANTHROPIC_MODEL
class TestToolChoiceSettings:
def test_ollama_forces_tool_choice(self, local_first):
settings = model_selector.get_tool_choice_settings()
assert settings.get("extra_body") == {"tool_choice": "required"}
def test_claude_uses_native_tool_choice(self, local_first, monkeypatch):
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", True)
settings = model_selector.get_tool_choice_settings()
assert not settings.get("extra_body")
class TestGetModelInfo:
def test_reports_ollama_primary(self, local_first):
info = model_selector.get_model_info()
assert info["backend"] == "ollama"
assert info["model"] == config.OLLAMA_DEFAULT_MODEL
assert info["ollama_available"] is True
assert info["claude_available"] is True
assert info["prefer_cloud"] is False
def test_reports_claude_when_ollama_down(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", False)
info = model_selector.get_model_info()
assert info["backend"] == "claude"
assert info["model"] == config.ANTHROPIC_MODEL
View File
+219
View File
@@ -0,0 +1,219 @@
"""
Wire-level contract tests for external service boundaries.
Each test sends the raw request the application code sends (no client
wrappers, no mocks) and asserts on the response shape, so boundary
breakage is caught directly instead of surfacing as agent misbehavior.
Semantics:
- Service unreachable -> skip (an outage is not a contract violation)
- Service reachable but wrong response shape -> fail
Run with: make test-contracts
"""
import json
import httpx
import pytest
from src.core.config import config
OLLAMA = str(config.OLLAMA_HOST).rstrip("/")
QDRANT = f"http://{config.QDRANT_HOST}:{config.QDRANT_PORT}"
SEARXNG = str(config.SEARXNG_HOST).rstrip("/")
CALCULATOR_TOOL = {
"type": "function",
"function": {
"name": "calculator",
"description": "Evaluate a math expression",
"parameters": {
"type": "object",
"properties": {"expression": {"type": "string"}},
"required": ["expression"],
},
},
}
async def _get_or_skip(url: str, service: str, timeout: float = 5.0) -> httpx.Response:
"""GET a URL, skipping the test if the service is unreachable."""
try:
async with httpx.AsyncClient(timeout=timeout) as client:
return await client.get(url)
except httpx.TransportError as e:
pytest.skip(f"{service} unreachable at {url}: {e}")
async def _post_or_skip(
url: str, service: str, payload: dict, timeout: float, headers: dict | None = None
) -> httpx.Response:
"""POST a payload, skipping the test if the service is unreachable."""
try:
async with httpx.AsyncClient(timeout=timeout) as client:
return await client.post(url, json=payload, headers=headers)
except httpx.TransportError as e:
pytest.skip(f"{service} unreachable at {url}: {e}")
@pytest.mark.contract
class TestOllamaContract:
"""Boundary: Ollama native API and its OpenAI-compat layer."""
async def test_tags_lists_configured_model(self):
# Mirrors check_ollama_health()
response = await _get_or_skip(f"{OLLAMA}/api/tags", "ollama")
assert response.status_code == 200
names = [m["name"] for m in response.json()["models"]]
model = config.OLLAMA_DEFAULT_MODEL
assert model in names or f"{model}:latest" in names, (
f"{model} not pulled; available: {names}"
)
async def test_generate_returns_plain_text(self):
# Mirrors StewardAgent._call_ollama()
response = await _post_or_skip(
f"{OLLAMA}/api/generate",
"ollama",
{
"model": config.OLLAMA_DEFAULT_MODEL,
"prompt": "Reply with the single word: pong",
"stream": False,
"options": {"temperature": 0.3, "top_p": 0.9},
},
timeout=config.OLLAMA_TIMEOUT,
)
assert response.status_code == 200
assert response.json()["response"].strip()
async def test_openai_compat_tool_calling(self):
# Mirrors the request PydanticAI's OpenAIChatModel sends for the
# orchestration phase, including the extra_body tool_choice.
response = await _post_or_skip(
f"{OLLAMA}/v1/chat/completions",
"ollama",
{
"model": config.OLLAMA_DEFAULT_MODEL,
"messages": [
{"role": "user", "content": "What is 6 * 7? Use the calculator."}
],
"tools": [CALCULATOR_TOOL],
"tool_choice": "required",
"stream": False,
},
timeout=config.OLLAMA_TIMEOUT,
)
assert response.status_code == 200
message = response.json()["choices"][0]["message"]
tool_calls = message.get("tool_calls")
assert tool_calls, f"model answered in text instead of calling the tool: {message}"
assert tool_calls[0]["function"]["name"] == "calculator"
arguments = json.loads(tool_calls[0]["function"]["arguments"])
assert "expression" in arguments
@pytest.mark.contract
class TestAnthropicContract:
"""Boundary: Anthropic Messages API (the Claude fallback backend)."""
HEADERS_KEY = "anthropic-version"
def _headers(self) -> dict:
if not config.ANTHROPIC_API_KEY:
pytest.skip("ANTHROPIC_API_KEY not configured")
return {
"x-api-key": config.ANTHROPIC_API_KEY,
"anthropic-version": "2023-06-01",
}
async def test_minimal_message_accepted(self):
# Mirrors check_claude_health(): tiny request, no sampling params
response = await _post_or_skip(
"https://api.anthropic.com/v1/messages",
"anthropic",
{
"model": config.ANTHROPIC_MODEL,
"max_tokens": 1,
"messages": [{"role": "user", "content": "hi"}],
},
timeout=30.0,
headers=self._headers(),
)
assert response.status_code == 200, response.text
async def test_temperature_rejected(self):
# Pins the Claude Sonnet 5+ contract that broke the Steward:
# sampling parameters are rejected with a 400 (and not billed).
response = await _post_or_skip(
"https://api.anthropic.com/v1/messages",
"anthropic",
{
"model": config.ANTHROPIC_MODEL,
"max_tokens": 1,
"messages": [{"role": "user", "content": "hi"}],
"temperature": 0.3,
},
timeout=30.0,
headers=self._headers(),
)
assert response.status_code == 400
assert "temperature" in response.text
@pytest.mark.contract
class TestQdrantContract:
"""Boundary: Qdrant REST API (Biographer's vector memory)."""
async def test_collections_endpoint(self):
response = await _get_or_skip(f"{QDRANT}/collections", "qdrant")
assert response.status_code == 200
assert "collections" in response.json()["result"]
@pytest.mark.contract
class TestSearxngContract:
"""Boundary: SearXNG JSON search API (web search tool)."""
async def test_json_search(self):
response = await _get_or_skip(
f"{SEARXNG}/search?q=test&format=json", "searxng", timeout=config.SEARXNG_TIMEOUT
)
assert response.status_code == 200
assert "results" in response.json()
@pytest.mark.contract
class TestLibraryDeskContract:
"""Boundary: library-desk research API (the Librarian's backend)."""
async def test_health(self):
host = getattr(config, "LIBRARY_DESK_HOST", None)
if not host:
pytest.skip("LIBRARY_DESK_HOST not configured")
response = await _get_or_skip(f"{str(host).rstrip('/')}/health", "library-desk")
assert response.status_code == 200
@pytest.mark.contract
class TestRedisContract:
"""Boundary: Redis on the configured memory DB."""
async def test_roundtrip(self):
import redis.asyncio as redis
client = redis.Redis(
host=config.REDIS_HOST,
port=config.REDIS_PORT,
db=config.REDIS_MEMORY_DB,
socket_connect_timeout=3,
)
try:
await client.ping()
except Exception as e:
pytest.skip(f"redis unreachable: {e}")
try:
await client.set("contract-test-key", "ok", ex=30)
assert await client.get("contract-test-key") == b"ok"
await client.delete("contract-test-key")
finally:
await client.aclose()