Compare commits

...
5 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Fable 5 11405e0acb chore: release v2.3.0
Build and Push / release (push) Successful in 25s
Build and Push / build (push) Successful in 4m34s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 23:54:32 +02:00
jpmschweitzerandClaude Fable 5 d2aeb8957b docs: document local-first backend, gemma4 gotchas, and contract tests
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Fable 5 d8c84d080c test: add wire-level service contract tests
tests/contracts sends the raw requests the code sends to Ollama (native API
and OpenAI-compat tool calling), Anthropic (including the pinned Sonnet 5
temperature-rejection contract), Qdrant, SearXNG, library-desk, and Redis.
Unreachable services skip; wrong response shapes fail. Run via
make test-contracts; excluded from the unit suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Fable 5 f03d41c698 fix: use dedicated tool-phase prompt for gemma4 orchestration
With the butler persona prompt attached, gemma4 reasons about calling the
calculator and then answers from memory with a different wrong product every
run; tool_choice=required via extra_body is advisory at best on Ollama's
OpenAI-compat layer. orchestrate_tool_calls() now uses a terse
TATLOCK_ORCHESTRATION_PROMPT; synthesize_from_results() keeps the persona,
so the user-visible voice is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Fable 5 033a1c01e8 feat: make Ollama/gemma4 the primary backend with Claude as fallback
Rolls back the claudification backend preference: PREFER_CLOUD_BACKEND now
defaults to false, resolve_backend() picks Ollama first and uses Claude when
explicitly preferred or when the new Ollama startup health check fails. The
Steward retries mid-request failures on the other backend in both directions.

Also hardens the fallback itself: Anthropic SDK imports are lazy so a broken
anthropic package degrades to Ollama-only instead of crashing at import time
(root cause of the production outage since April), anthropic is pinned to a
pydantic-ai-1.27-compatible range, ANTHROPIC_MODEL defaults to claude-sonnet-5
(sonnet-4-20250514 retired 2026-06-15), sampling parameters are stripped from
Claude calls (Sonnet 5 rejects them), and the Steward timeout is configurable
(STEWARD_TIMEOUT, default 60s) since gemma4 needs ~35s warm for analysis.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
20 changed files with 611 additions and 88 deletions
+10 -8
View File
@@ -8,17 +8,19 @@ API_HOST=0.0.0.0
API_PORT=8000
API_PREFIX=/v1
# Anthropic Configuration (Claude - preferred backend)
# Set ANTHROPIC_API_KEY to enable Claude as the default backend
# Without an API key, Tatlock uses Ollama exclusively
# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
ANTHROPIC_MODEL=claude-sonnet-4-20250514
PREFER_CLOUD_BACKEND=true
# Ollama Configuration (local fallback when Claude unavailable)
# Ollama Configuration (local - primary backend)
OLLAMA_HOST=http://localhost:11434
OLLAMA_DEFAULT_MODEL=gemma4:e2b
OLLAMA_TIMEOUT=120
STEWARD_TIMEOUT=60
# Anthropic Configuration (Claude - cloud fallback)
# Set ANTHROPIC_API_KEY to keep the Claude fallback available: it is used
# automatically when Ollama is down, or exclusively when PREFER_CLOUD_BACKEND=true
# Without an API key, Tatlock uses Ollama only
# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
ANTHROPIC_MODEL=claude-sonnet-5
PREFER_CLOUD_BACKEND=false
# SearXNG Configuration
SEARXNG_HOST=http://localhost:8087
+21
View File
@@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.3.0] - 2026-07-13
### Changed
- **Local-first backend (claudification rollback)** - Ollama/gemma4 is now the primary backend; Claude remains as fallback. `PREFER_CLOUD_BACKEND` defaults to `false`, Claude is used automatically when the Ollama startup health check fails, and the Steward retries mid-request failures on the other backend in both directions
- **Default Claude model `claude-sonnet-5`** - `claude-sonnet-4-20250514` was retired by Anthropic on 2026-06-15 and would 404, leaving the fallback dead
- **Dedicated orchestration prompt** - `orchestrate_tool_calls()` now uses a terse tool-execution prompt (`TATLOCK_ORCHESTRATION_PROMPT`); the butler persona prompt suppressed gemma4 tool calling (the model reasoned about the calculator, then answered from memory with wrong arithmetic). Synthesis keeps the persona prompt, so user-visible voice is unchanged
### Fixed
- **Startup crash with broken anthropic package** - Anthropic SDK imports in the model selector are now lazy, so an incompatible `anthropic` install degrades to Ollama-only operation instead of crashing the app at import time (root cause of the production outage since April)
- **Claude Sonnet 5 rejects sampling parameters** - removed `temperature` from the Steward's direct Claude call and made the Housekeeper's temperature setting backend-conditional via `get_sampling_settings()`
- **Pin `anthropic>=0.77,<1.0`** - the April image resolved an anthropic version incompatible with pydantic-ai 1.27
- **Steward timeout configurable** - new `STEWARD_TIMEOUT` (default 60s) replaces the hardcoded 30s, which gemma4 chronically exceeded (~35s warm analysis), causing every request to fail or fall back
### Added
- **Ollama startup health check** - verifies the server is reachable and `OLLAMA_DEFAULT_MODEL` is pulled; feeds backend resolution and `get_model_info()`
- **Contract tests** (`tests/contracts/`, `make test-contracts`) - wire-level tests that send the raw requests the code sends to Ollama (native + OpenAI-compat tool calling), Anthropic (including the pinned temperature-rejection contract), Qdrant, SearXNG, library-desk, and Redis; unreachable services skip, wrong response shapes fail
- **Backend resolution unit tests** (`tests/anthropic/`)
## [2.2.0] - 2026-04-04
### Changed
+6 -3
View File
@@ -8,6 +8,7 @@ Claude Code-specific notes for this project. For general development instruction
make setup # Create venv and install all dependencies
make test # Unit tests (no external services)
make test-integration # Integration tests (needs Claude/Ollama)
make test-contracts # Wire-level contract tests against live service boundaries
make run # Start dev server on port 8777
make lint # Ruff linter + formatter check
make typecheck # Mypy
@@ -18,13 +19,15 @@ Dependencies are in `pyproject.toml` (`[project.dependencies]` and `[project.opt
## Critical Gotchas
**ASGITransport does NOT trigger FastAPI lifespan events.** The session-scoped `_initialize_app` fixture in `tests/conftest.py` calls `initialize_application()` explicitly via `asyncio.run()`. Without this, `check_claude_health()` never runs and `_claude_available` stays `None`, causing all tests to silently fall back to Ollama.
**ASGITransport does NOT trigger FastAPI lifespan events.** The session-scoped `_initialize_app` fixture in `tests/conftest.py` calls `initialize_application()` explicitly via `asyncio.run()`. Without this, the Ollama/Claude health checks never run: `_ollama_available` stays `None` (treated as available, so requests go to Ollama) and `_claude_available` stays `None` (treated as unavailable, so the Claude fallback never engages).
**AsyncIO scope mismatch.** `asyncio_default_fixture_loop_scope = function` is set in `pyproject.toml`. Session-scoped async fixtures cause `ScopeMismatch` errors. The fix is to use a sync fixture with `asyncio.run()` for session-scoped initialization.
**Ollama is unreliable for tool calling.** `mistral-nemo` on Ollama often does mental math instead of calling calculator tools, and frequently gets wrong answers. Claude reliably calls tools. If integration tests give wrong math answers, check which backend is actually being used.
**The butler persona prompt suppresses local-model tool calling.** With `TATLOCK_SYSTEM_PROMPT` attached, gemma4 reasons about calling the calculator, then answers from memory with wrong arithmetic (a different wrong product each run). `orchestrate_tool_calls()` therefore uses the terse `TATLOCK_ORCHESTRATION_PROMPT`; the persona is applied in `synthesize_from_results()`. Do not reattach the persona prompt to a tool-phase agent. `tool_choice: "required"` via extra_body does NOT force Ollama to call tools — it is advisory at best.
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config. Ollama on tower-of-joy can be slow, especially on first request.
**Claude Sonnet 5+ rejects sampling parameters.** `temperature`/`top_p`/`top_k` return a 400. Use `get_sampling_settings()` from the model selector instead of passing `ModelSettings(temperature=...)` directly to agents that can run on the Claude fallback. The contract test suite pins this (`make test-contracts`).
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). The full local Steward → orchestrate → synthesize flow takes ~2 minutes on gemma4. Steward analysis alone needs ~35s warm — `STEWARD_TIMEOUT` defaults to 60s.
**`get_benchmark_store` does not exist.** The benchmarking module (`src/core/benchmarks.py`) was never implemented. `scripts/benchmark_analysis.py` also references it and is broken. Do not add mocks for it in tests.
+5 -2
View File
@@ -1,4 +1,4 @@
.PHONY: help setup run test test-unit test-integration lint typecheck clean
.PHONY: help setup run test test-unit test-integration test-contracts lint typecheck clean
VENV := .venv
PYTHON := $(VENV)/bin/python
@@ -29,13 +29,16 @@ run: ## Start the development server on port 8777
$(UVICORN) src.main:app --reload --host $(HOST) --port $(PORT) 2>&1 | tee build/logs/server.log
test: ## Run unit tests (no external services needed)
$(PYTEST) --ignore=tests/e2e --ignore=tests/integration
$(PYTEST) --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts
test-unit: test ## Alias for test
test-integration: ## Run integration tests (needs Claude/Ollama)
$(PYTEST) tests/agents/test_tatlock_agent.py -v
test-contracts: ## Wire-level contract tests against live service boundaries
$(PYTEST) tests/contracts -v --no-cov
lint: ## Run ruff linter and formatter check
$(RUFF) check src tests
$(RUFF) format --check src tests
+15 -7
View File
@@ -58,7 +58,7 @@ A privacy-first, offline-capable personal assistant system that coordinates spec
- Error triggers for testing (rate_limit, context_overflow)
- **Tatlock**: Real PydanticAI agent with butler personality
- **LLM Backend**: Ollama (mistral-nemo:latest by default)
- **LLM Backend**: Ollama (gemma4:e2b by default, local-first) with optional Claude fallback
- **Personality**: Witty British butler, research-oriented
- **Core Tools**:
- **Calculator**: Safe mathematical expression evaluation
@@ -74,7 +74,7 @@ A privacy-first, offline-capable personal assistant system that coordinates spec
- Python 3.12+ (Python 3.12.11 recommended)
- **External Services** (must be running separately):
- **Ollama**: LLM inference (mistral-nemo:latest, nomic-embed-text)
- **Ollama**: LLM inference (gemma4:e2b, nomic-embed-text)
- **Redis**: Caching and session memory
- **Qdrant**: Vector storage for The Biographer's memory
- **SearXNG**: Web search (optional)
@@ -264,7 +264,10 @@ Interactive documentation available at:
pytest
# Run unit tests only (no external services needed)
pytest --ignore=tests/e2e --ignore=tests/integration
pytest --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts
# Wire-level contract tests against live service boundaries
make test-contracts
# Run with coverage
pytest --cov=src --cov-report=term-missing
@@ -303,12 +306,17 @@ Create a `.env` file for custom configuration:
API_HOST=0.0.0.0
API_PORT=8000
# Ollama Configuration
# Ollama Configuration (primary backend)
OLLAMA_HOST=http://localhost:11434
OLLAMA_DEFAULT_MODEL=mistral-nemo:latest
OLLAMA_DEFAULT_MODEL=gemma4:e2b
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
OLLAMA_TIMEOUT=120
# Claude fallback (optional; used when Ollama is down or PREFER_CLOUD_BACKEND=true)
# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
ANTHROPIC_MODEL=claude-sonnet-5
PREFER_CLOUD_BACKEND=false
# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
@@ -427,8 +435,8 @@ For LLM agent development guidelines and architectural decisions, see [AGENTS.md
## Version
Current version: **1.3.2** - Biographer tool type hints fix
Current version: see [CHANGELOG.md](CHANGELOG.md)
---
**Note**: Tatlock is a production-ready homelab butler. All household staff use PydanticAI with Ollama for local LLM inference.
**Note**: Tatlock is a production-ready homelab butler. All household staff use PydanticAI with local Ollama inference (gemma4), with an optional Claude cloud fallback.
+1 -1
View File
@@ -3,7 +3,7 @@
## Overview
Tatlock uses a bidirectional Claude architecture:
- **Scenario A**: Tatlock powered by Claude backend (with Ollama fallback) — **COMPLETE**
- **Scenario A**: Tatlock powered by Claude backend (with Ollama fallback) — **COMPLETE**, then **rolled back to local-first**: Ollama/gemma4 is primary, Claude is retained as fallback (`PREFER_CLOUD_BACKEND=false`)
- **Scenario B**: Tatlock exposed as MCP server for external Claude instances — **OPEN**
- **Scenario C**: Offline operation via Ollama — **COMPLETE**
+1 -1
View File
@@ -12,7 +12,7 @@ This document tracks open/planned work. Completed phases have been removed.
- Household staff: Tatlock (Butler), Steward, Librarian, Biographer
- Core tools: Calculator, Date/Time, Web search (SearXNG)
- Memory system: Qdrant (vector), Redis (session cache), multi-tenancy via ContextVar
- Dual backend: Claude (preferred) + Ollama (fallback)
- Dual backend: Ollama/gemma4 (primary) + Claude (fallback)
- 439 tests with good coverage
---
+3 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "tatlock"
version = "2.2.0"
version = "2.3.0"
description = "OpenAI-compatible API with Ollama backend"
requires-python = ">=3.12"
dependencies = [
@@ -13,6 +13,7 @@ dependencies = [
"pydantic>=2.11,<2.13",
"pydantic-settings>=2.12,<2.13",
"pydantic-ai-slim[openai,anthropic]>=1.27,<1.28",
"anthropic>=0.77,<1.0",
"httpx>=0.28,<0.29",
"sse-starlette>=3.0,<3.1",
"python-dotenv>=1.2,<1.3",
@@ -46,6 +47,7 @@ markers = [
"unit: Unit tests",
"integration: Integration tests",
"slow: Slow running tests",
"contract: Wire-level contract tests against live service boundaries",
]
addopts = [
"--verbose",
+6 -6
View File
@@ -204,13 +204,13 @@ async def run_housekeeper(
)
try:
# Use temperature 0.1 for slight exploration
from pydantic_ai.settings import ModelSettings
# Temperature 0.1 for slight exploration (skipped on Claude backend)
from src.anthropic.model_selector import get_sampling_settings
result = await agent.run(
prompt,
message_history=message_history,
model_settings=ModelSettings(temperature=0.1),
model_settings=get_sampling_settings(0.1),
)
logger.info(
@@ -266,13 +266,13 @@ async def run_housekeeper_stream(
)
try:
# Use temperature 0.1 for slight exploration
from pydantic_ai.settings import ModelSettings
# Temperature 0.1 for slight exploration (skipped on Claude backend)
from src.anthropic.model_selector import get_sampling_settings
async with agent.run_stream(
prompt,
message_history=message_history,
model_settings=ModelSettings(temperature=0.1),
model_settings=get_sampling_settings(0.1),
) as response:
async for delta in response.stream_text(delta=True):
yield delta
+28 -14
View File
@@ -11,7 +11,7 @@ Uses plain text output (not JSON) for reliability. Supports both Claude
import httpx
from typing import Optional
from src.anthropic.model_selector import is_claude_available, get_model_info
from src.anthropic.model_selector import get_model_info, is_claude_available, resolve_backend
from src.core.config import config
from src.core.household_registry import get_household_registry
from src.core.logging_config import get_logger
@@ -113,18 +113,19 @@ class StewardAgent:
def __init__(self):
"""Initialize Steward with backend selection based on availability."""
# Ollama config (fallback)
# Ollama config (primary)
self.ollama_host = str(config.OLLAMA_HOST).rstrip('/')
self.ollama_model = config.OLLAMA_DEFAULT_MODEL
# Claude config (preferred)
# Claude config (fallback)
self.claude_model = config.ANTHROPIC_MODEL
self._anthropic_client = None
# Determine which backend to use
self._use_claude = config.PREFER_CLOUD_BACKEND and is_claude_available()
# Determine which backend to use (Ollama-first, Claude when
# preferred via config or when Ollama is down)
self._use_claude = resolve_backend() == "claude"
self.timeout = 30.0 # 30 second timeout for analysis
self.timeout = float(config.STEWARD_TIMEOUT)
model_info = get_model_info()
logger.info(
@@ -145,12 +146,12 @@ class StewardAgent:
"""Call Claude API directly for plain text generation."""
client = self._get_anthropic_client()
# No temperature: rejected by Claude Sonnet 5+ (sampling params deprecated)
response = await client.messages.create(
model=self.claude_model,
max_tokens=1024,
system=system_prompt,
messages=[{"role": "user", "content": user_message}],
temperature=0.3, # Lower = more consistent
)
return response.content[0].text.strip()
@@ -227,20 +228,33 @@ class StewardAgent:
return analysis_text
except Exception as e:
# If Claude fails, try Ollama as fallback
# Mid-request fallback: retry on the other backend when possible
if self._use_claude:
logger.warning(
"steward_claude_fallback",
error=str(e),
)
analysis_text = await self._call_ollama(prompt)
logger.debug(
"steward_analysis_received",
backend="ollama_fallback",
text_preview=analysis_text[:150],
fallback_backend = "ollama_fallback"
elif is_claude_available():
logger.warning(
"steward_ollama_fallback",
error=str(e),
)
return analysis_text
raise
analysis_text = await self._call_claude(
system_prompt="You are the Steward of the household, advising the Butler (Tatlock) on which capabilities to use. Be concise and specific.",
user_message=prompt,
)
fallback_backend = "claude_fallback"
else:
raise
logger.debug(
"steward_analysis_received",
backend=fallback_backend,
text_preview=analysis_text[:150],
)
return analysis_text
# Global Steward instance
+18 -2
View File
@@ -129,6 +129,22 @@ or
"""
# Tool-phase prompt for orchestrate_tool_calls(). The butler personality prompt
# suppresses tool calling on small local models (gemma4 reasons about the tool,
# then answers from memory with wrong arithmetic), so the orchestration phase
# uses a terse operator prompt; synthesize_from_results() applies the persona.
TATLOCK_ORCHESTRATION_PROMPT = """You are the tool-execution phase of Tatlock, \
a butler assistant. Your only job is to gather accurate results by calling the \
provided tools.
- ALWAYS use tools for the task - never answer from memory and never do mental math.
- Mathematics: call the calculate tool, even for trivial arithmetic.
- Dates and times: call the date/time tools, never guess.
- When the instructions say DELEGATE to an agent, call the matching delegate_to_* tool.
- After the tool results arrive, reply with a one-line factual summary of the results. \
A later step writes the polished reply, so do not add personality."""
class TatlockAgent(AgentInterface):
"""
Tatlock - The Butler agent using PydanticAI with Ollama.
@@ -654,10 +670,10 @@ class TatlockAgent(AgentInterface):
# Create a fresh agent instance with scoped tools only
model = get_model()
# Create agent with scoped tools
# Create agent with scoped tools, using the tool-phase prompt
scoped_agent = Agent(
model,
system_prompt=TATLOCK_SYSTEM_PROMPT,
system_prompt=TATLOCK_ORCHESTRATION_PROMPT,
tools=scoped_tools,
)
+8 -1
View File
@@ -1,19 +1,26 @@
"""
Anthropic/Claude integration module.
Provides model selection with automatic fallback between Claude and Ollama.
Provides model selection with Ollama as primary backend and Claude
as the cloud fallback.
"""
from src.anthropic.model_selector import (
check_claude_health,
check_ollama_health,
get_model,
get_tool_choice_settings,
is_claude_available,
is_ollama_available,
resolve_backend,
)
__all__ = [
"check_claude_health",
"check_ollama_health",
"get_model",
"get_tool_choice_settings",
"is_claude_available",
"is_ollama_available",
"resolve_backend",
]
+155 -31
View File
@@ -1,23 +1,84 @@
"""
Model selector for Claude/Ollama backend switching.
Model selector for Ollama/Claude backend switching.
Provides automatic model selection with Claude as preferred backend
and Ollama as offline fallback.
Provides automatic model selection with Ollama as the primary local backend
and Claude as the cloud fallback. Claude is used when PREFER_CLOUD_BACKEND
is enabled, or automatically when Ollama is unavailable at startup.
The Anthropic SDK is imported lazily so a missing or broken `anthropic`
package degrades to Ollama-only operation instead of crashing the app.
"""
from typing import Union
from __future__ import annotations
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.anthropic import AnthropicProvider
from typing import TYPE_CHECKING
import httpx
from src.core.config import config
from src.core.logging_config import get_logger
if TYPE_CHECKING:
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.settings import ModelSettings
logger = get_logger(__name__)
# Cached health check result (set once at startup)
# Cached health check results (set once at startup)
_claude_available: bool | None = None
_ollama_available: bool | None = None
async def check_ollama_health() -> bool:
"""
Check if the Ollama server is reachable and has the configured model.
This should be called once at application startup.
The result is cached in `_ollama_available`.
Returns:
True if Ollama is reachable and OLLAMA_DEFAULT_MODEL is pulled.
"""
global _ollama_available
host = str(config.OLLAMA_HOST).rstrip("/")
model = config.OLLAMA_DEFAULT_MODEL
try:
async with httpx.AsyncClient(timeout=5.0) as client:
response = await client.get(f"{host}/api/tags")
response.raise_for_status()
names = [m.get("name", "") for m in response.json().get("models", [])]
if model in names or f"{model}:latest" in names:
_ollama_available = True
logger.info(
"ollama_health_check_passed",
host=host,
model=model,
)
return True
_ollama_available = False
logger.warning(
"ollama_health_check_failed",
reason="model_not_pulled",
host=host,
model=model,
hint=f"run `ollama pull {model}`",
)
return False
except Exception as e:
_ollama_available = False
logger.warning(
"ollama_health_check_failed",
reason="server_unreachable",
host=host,
error=str(e),
)
return False
async def check_claude_health() -> bool:
@@ -85,46 +146,93 @@ def is_claude_available() -> bool:
return _claude_available is True
def get_model(prefer_cloud: bool | None = None) -> Union[AnthropicModel, OpenAIChatModel]:
def is_ollama_available() -> bool:
"""
Check if Ollama is available (from cached health check result).
Returns:
False only if the startup health check confirmed Ollama is down.
Unknown (check not run yet) counts as available so that contexts
without lifespan events keep the local-first behavior.
"""
return _ollama_available is not False
def resolve_backend(prefer_cloud: bool | None = None) -> str:
"""
Resolve which backend should serve requests.
Ollama is the primary backend. Claude is used when explicitly
preferred via PREFER_CLOUD_BACKEND, or as automatic fallback
when the startup health check found Ollama down.
Args:
prefer_cloud: Override config.PREFER_CLOUD_BACKEND for this call.
Returns:
"claude" or "ollama".
"""
use_cloud = prefer_cloud if prefer_cloud is not None else config.PREFER_CLOUD_BACKEND
if use_cloud and is_claude_available():
return "claude"
if not is_ollama_available() and is_claude_available():
logger.warning(
"backend_fallback_to_claude",
reason="ollama_unavailable",
)
return "claude"
return "ollama"
def get_model(prefer_cloud: bool | None = None) -> AnthropicModel | OpenAIChatModel:
"""
Get the best available model.
Returns Claude if available and preferred, otherwise Ollama.
Returns Ollama unless Claude is preferred (or Ollama is down).
Args:
prefer_cloud: Override config.PREFER_CLOUD_BACKEND for this call.
If None, uses the config value.
Returns:
PydanticAI model instance (AnthropicModel or OpenAIChatModel).
PydanticAI model instance (OpenAIChatModel or AnthropicModel).
Example:
>>> model = get_model()
>>> agent = Agent(model, system_prompt="...")
"""
# Determine preference
use_cloud = prefer_cloud if prefer_cloud is not None else config.PREFER_CLOUD_BACKEND
if resolve_backend(prefer_cloud) == "claude":
try:
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
# Use Claude if available and preferred
if use_cloud and is_claude_available():
logger.debug(
"model_selected",
backend="claude",
model=config.ANTHROPIC_MODEL,
)
return AnthropicModel(
model_name=config.ANTHROPIC_MODEL,
provider=AnthropicProvider(api_key=config.ANTHROPIC_API_KEY),
)
logger.debug(
"model_selected",
backend="claude",
model=config.ANTHROPIC_MODEL,
)
return AnthropicModel(
model_name=config.ANTHROPIC_MODEL,
provider=AnthropicProvider(api_key=config.ANTHROPIC_API_KEY),
)
except ImportError as e:
logger.error(
"claude_backend_import_failed",
error=str(e),
hint="anthropic package missing or incompatible; using Ollama",
)
from pydantic_ai.models.openai import OpenAIChatModel
# Fall back to Ollama
from src.ollama.provider import get_ollama_provider
logger.debug(
"model_selected",
backend="ollama",
model=config.OLLAMA_DEFAULT_MODEL,
reason="fallback" if use_cloud else "preferred_local",
)
return OpenAIChatModel(
model_name=config.OLLAMA_DEFAULT_MODEL,
@@ -132,7 +240,7 @@ def get_model(prefer_cloud: bool | None = None) -> Union[AnthropicModel, OpenAIC
)
def get_tool_choice_settings() -> 'ModelSettings':
def get_tool_choice_settings() -> ModelSettings:
"""
Get model_settings for forcing tool calls on the first request.
@@ -141,7 +249,7 @@ def get_tool_choice_settings() -> 'ModelSettings':
"""
from pydantic_ai.settings import ModelSettings
if is_claude_available() and config.PREFER_CLOUD_BACKEND:
if resolve_backend() == "claude":
# PydanticAI's Anthropic model handles tool_choice internally
return ModelSettings()
else:
@@ -149,6 +257,20 @@ def get_tool_choice_settings() -> 'ModelSettings':
return ModelSettings(extra_body={"tool_choice": "required"})
def get_sampling_settings(temperature: float) -> ModelSettings:
"""
Get model_settings with a sampling temperature where the backend allows it.
Ollama accepts a temperature; Claude Sonnet 5+ rejects sampling
parameters, so the Claude backend gets empty settings.
"""
from pydantic_ai.settings import ModelSettings
if resolve_backend() == "claude":
return ModelSettings()
return ModelSettings(temperature=temperature)
def get_model_info() -> dict:
"""
Get information about the current model configuration.
@@ -158,12 +280,14 @@ def get_model_info() -> dict:
Returns:
Dict with backend, model name, and availability info.
"""
use_cloud = config.PREFER_CLOUD_BACKEND and is_claude_available()
backend = resolve_backend()
return {
"backend": "claude" if use_cloud else "ollama",
"model": config.ANTHROPIC_MODEL if use_cloud else config.OLLAMA_DEFAULT_MODEL,
"backend": backend,
"model": config.ANTHROPIC_MODEL if backend == "claude" else config.OLLAMA_DEFAULT_MODEL,
"claude_available": is_claude_available(),
"claude_configured": bool(config.ANTHROPIC_API_KEY),
"ollama_available": is_ollama_available(),
"ollama_model": config.OLLAMA_DEFAULT_MODEL,
"prefer_cloud": config.PREFER_CLOUD_BACKEND,
}
+11 -7
View File
@@ -64,21 +64,21 @@ class Config(BaseSettings):
API_PORT: int = Field(default=8000, description="API port")
API_PREFIX: str = Field(default="/v1", description="API route prefix")
# Anthropic Configuration (Claude - preferred backend)
# Anthropic Configuration (Claude - cloud fallback)
ANTHROPIC_API_KEY: str | None = Field(
default=None,
description="Anthropic API key for Claude access"
description="Anthropic API key for the Claude fallback backend"
)
ANTHROPIC_MODEL: str = Field(
default="claude-sonnet-4-20250514",
description="Claude model to use"
default="claude-sonnet-5",
description="Claude model for the fallback backend"
)
PREFER_CLOUD_BACKEND: bool = Field(
default=True,
description="Prefer Claude over Ollama when available"
default=False,
description="Prefer Claude over Ollama (default: local-first)"
)
# Ollama Configuration (local fallback)
# Ollama Configuration (local - primary backend)
OLLAMA_HOST: HttpUrl = Field(
default="http://localhost:11434",
description="Ollama server URL"
@@ -91,6 +91,10 @@ class Config(BaseSettings):
default=120,
description="Ollama request timeout in seconds"
)
STEWARD_TIMEOUT: int = Field(
default=60,
description="Steward analysis timeout in seconds (gemma4 needs ~35s warm)"
)
STREAM_TIMEOUT: int = Field(
default=20,
description="Timeout for each streaming turn in seconds"
+9 -3
View File
@@ -9,7 +9,11 @@ from src.agents.biographer import register_biographer
from src.agents.housekeeper import register_housekeeper
from src.agents.librarian import register_librarian
from src.agents.tatlock_core import TATLOCK_CORE_CAPABILITY, tatlock_core_tools
from src.anthropic.model_selector import check_claude_health, get_model_info
from src.anthropic.model_selector import (
check_claude_health,
check_ollama_health,
get_model_info,
)
from src.core.household_registry import get_household_registry
from src.core.logging_config import get_logger
@@ -87,7 +91,7 @@ async def initialize_application():
Initialize the application.
Performs all startup tasks:
1. Check Claude API health (for backend selection)
1. Check Ollama (primary) and Claude (fallback) health for backend selection
2. Register household members
3. (Future) Initialize connections
@@ -95,13 +99,15 @@ async def initialize_application():
"""
logger.info("application_initialization_starting")
# Check Claude API health for backend selection
# Check backend health: Ollama is primary, Claude is the fallback
await check_ollama_health()
await check_claude_health()
model_info = get_model_info()
logger.info(
"model_backend_configured",
backend=model_info["backend"],
model=model_info["model"],
ollama_available=model_info["ollama_available"],
claude_available=model_info["claude_available"],
)
+3 -1
View File
@@ -410,10 +410,12 @@ async def test_tatlock_ollama_fallback(async_client: AsyncClient):
"stream": False
}
# 300s: this test forbids the Claude rescue, and the full local
# Steward -> orchestrate -> synthesize flow on gemma4 exceeds 120s
response = await async_client.post(
"/v1/chat/completions",
json=request_data,
timeout=120.0
timeout=300.0
)
assert response.status_code == 200
View File
+92
View File
@@ -0,0 +1,92 @@
"""
Unit tests for backend selection (Ollama primary, Claude fallback).
These tests set the cached health-check globals directly so they are
deterministic regardless of which services are reachable.
"""
import pytest
from src.anthropic import model_selector
from src.core.config import config
@pytest.fixture
def local_first(monkeypatch):
"""Baseline: local-first config, both backends healthy."""
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", False)
monkeypatch.setattr(config, "ANTHROPIC_API_KEY", "sk-test-fake")
monkeypatch.setattr(model_selector, "_claude_available", True)
monkeypatch.setattr(model_selector, "_ollama_available", True)
class TestResolveBackend:
def test_default_is_ollama(self, local_first):
assert model_selector.resolve_backend() == "ollama"
def test_prefer_cloud_config_selects_claude(self, local_first, monkeypatch):
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", True)
assert model_selector.resolve_backend() == "claude"
def test_prefer_cloud_override_selects_claude(self, local_first):
assert model_selector.resolve_backend(prefer_cloud=True) == "claude"
def test_prefer_cloud_without_claude_falls_back_to_ollama(self, local_first, monkeypatch):
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", True)
monkeypatch.setattr(model_selector, "_claude_available", False)
assert model_selector.resolve_backend() == "ollama"
def test_ollama_down_falls_back_to_claude(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", False)
assert model_selector.resolve_backend() == "claude"
def test_ollama_down_without_claude_stays_ollama(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", False)
monkeypatch.setattr(model_selector, "_claude_available", False)
assert model_selector.resolve_backend() == "ollama"
def test_unknown_ollama_state_counts_as_available(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", None)
assert model_selector.resolve_backend() == "ollama"
class TestGetModel:
def test_ollama_backend_returns_openai_chat_model(self, local_first):
from pydantic_ai.models.openai import OpenAIChatModel
model = model_selector.get_model()
assert isinstance(model, OpenAIChatModel)
assert model.model_name == config.OLLAMA_DEFAULT_MODEL
def test_claude_backend_returns_anthropic_model(self, local_first):
from pydantic_ai.models.anthropic import AnthropicModel
model = model_selector.get_model(prefer_cloud=True)
assert isinstance(model, AnthropicModel)
assert model.model_name == config.ANTHROPIC_MODEL
class TestToolChoiceSettings:
def test_ollama_forces_tool_choice(self, local_first):
settings = model_selector.get_tool_choice_settings()
assert settings.get("extra_body") == {"tool_choice": "required"}
def test_claude_uses_native_tool_choice(self, local_first, monkeypatch):
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", True)
settings = model_selector.get_tool_choice_settings()
assert not settings.get("extra_body")
class TestGetModelInfo:
def test_reports_ollama_primary(self, local_first):
info = model_selector.get_model_info()
assert info["backend"] == "ollama"
assert info["model"] == config.OLLAMA_DEFAULT_MODEL
assert info["ollama_available"] is True
assert info["claude_available"] is True
assert info["prefer_cloud"] is False
def test_reports_claude_when_ollama_down(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", False)
info = model_selector.get_model_info()
assert info["backend"] == "claude"
assert info["model"] == config.ANTHROPIC_MODEL
View File
+219
View File
@@ -0,0 +1,219 @@
"""
Wire-level contract tests for external service boundaries.
Each test sends the raw request the application code sends (no client
wrappers, no mocks) and asserts on the response shape, so boundary
breakage is caught directly instead of surfacing as agent misbehavior.
Semantics:
- Service unreachable -> skip (an outage is not a contract violation)
- Service reachable but wrong response shape -> fail
Run with: make test-contracts
"""
import json
import httpx
import pytest
from src.core.config import config
OLLAMA = str(config.OLLAMA_HOST).rstrip("/")
QDRANT = f"http://{config.QDRANT_HOST}:{config.QDRANT_PORT}"
SEARXNG = str(config.SEARXNG_HOST).rstrip("/")
CALCULATOR_TOOL = {
"type": "function",
"function": {
"name": "calculator",
"description": "Evaluate a math expression",
"parameters": {
"type": "object",
"properties": {"expression": {"type": "string"}},
"required": ["expression"],
},
},
}
async def _get_or_skip(url: str, service: str, timeout: float = 5.0) -> httpx.Response:
"""GET a URL, skipping the test if the service is unreachable."""
try:
async with httpx.AsyncClient(timeout=timeout) as client:
return await client.get(url)
except httpx.TransportError as e:
pytest.skip(f"{service} unreachable at {url}: {e}")
async def _post_or_skip(
url: str, service: str, payload: dict, timeout: float, headers: dict | None = None
) -> httpx.Response:
"""POST a payload, skipping the test if the service is unreachable."""
try:
async with httpx.AsyncClient(timeout=timeout) as client:
return await client.post(url, json=payload, headers=headers)
except httpx.TransportError as e:
pytest.skip(f"{service} unreachable at {url}: {e}")
@pytest.mark.contract
class TestOllamaContract:
"""Boundary: Ollama native API and its OpenAI-compat layer."""
async def test_tags_lists_configured_model(self):
# Mirrors check_ollama_health()
response = await _get_or_skip(f"{OLLAMA}/api/tags", "ollama")
assert response.status_code == 200
names = [m["name"] for m in response.json()["models"]]
model = config.OLLAMA_DEFAULT_MODEL
assert model in names or f"{model}:latest" in names, (
f"{model} not pulled; available: {names}"
)
async def test_generate_returns_plain_text(self):
# Mirrors StewardAgent._call_ollama()
response = await _post_or_skip(
f"{OLLAMA}/api/generate",
"ollama",
{
"model": config.OLLAMA_DEFAULT_MODEL,
"prompt": "Reply with the single word: pong",
"stream": False,
"options": {"temperature": 0.3, "top_p": 0.9},
},
timeout=config.OLLAMA_TIMEOUT,
)
assert response.status_code == 200
assert response.json()["response"].strip()
async def test_openai_compat_tool_calling(self):
# Mirrors the request PydanticAI's OpenAIChatModel sends for the
# orchestration phase, including the extra_body tool_choice.
response = await _post_or_skip(
f"{OLLAMA}/v1/chat/completions",
"ollama",
{
"model": config.OLLAMA_DEFAULT_MODEL,
"messages": [
{"role": "user", "content": "What is 6 * 7? Use the calculator."}
],
"tools": [CALCULATOR_TOOL],
"tool_choice": "required",
"stream": False,
},
timeout=config.OLLAMA_TIMEOUT,
)
assert response.status_code == 200
message = response.json()["choices"][0]["message"]
tool_calls = message.get("tool_calls")
assert tool_calls, f"model answered in text instead of calling the tool: {message}"
assert tool_calls[0]["function"]["name"] == "calculator"
arguments = json.loads(tool_calls[0]["function"]["arguments"])
assert "expression" in arguments
@pytest.mark.contract
class TestAnthropicContract:
"""Boundary: Anthropic Messages API (the Claude fallback backend)."""
HEADERS_KEY = "anthropic-version"
def _headers(self) -> dict:
if not config.ANTHROPIC_API_KEY:
pytest.skip("ANTHROPIC_API_KEY not configured")
return {
"x-api-key": config.ANTHROPIC_API_KEY,
"anthropic-version": "2023-06-01",
}
async def test_minimal_message_accepted(self):
# Mirrors check_claude_health(): tiny request, no sampling params
response = await _post_or_skip(
"https://api.anthropic.com/v1/messages",
"anthropic",
{
"model": config.ANTHROPIC_MODEL,
"max_tokens": 1,
"messages": [{"role": "user", "content": "hi"}],
},
timeout=30.0,
headers=self._headers(),
)
assert response.status_code == 200, response.text
async def test_temperature_rejected(self):
# Pins the Claude Sonnet 5+ contract that broke the Steward:
# sampling parameters are rejected with a 400 (and not billed).
response = await _post_or_skip(
"https://api.anthropic.com/v1/messages",
"anthropic",
{
"model": config.ANTHROPIC_MODEL,
"max_tokens": 1,
"messages": [{"role": "user", "content": "hi"}],
"temperature": 0.3,
},
timeout=30.0,
headers=self._headers(),
)
assert response.status_code == 400
assert "temperature" in response.text
@pytest.mark.contract
class TestQdrantContract:
"""Boundary: Qdrant REST API (Biographer's vector memory)."""
async def test_collections_endpoint(self):
response = await _get_or_skip(f"{QDRANT}/collections", "qdrant")
assert response.status_code == 200
assert "collections" in response.json()["result"]
@pytest.mark.contract
class TestSearxngContract:
"""Boundary: SearXNG JSON search API (web search tool)."""
async def test_json_search(self):
response = await _get_or_skip(
f"{SEARXNG}/search?q=test&format=json", "searxng", timeout=config.SEARXNG_TIMEOUT
)
assert response.status_code == 200
assert "results" in response.json()
@pytest.mark.contract
class TestLibraryDeskContract:
"""Boundary: library-desk research API (the Librarian's backend)."""
async def test_health(self):
host = getattr(config, "LIBRARY_DESK_HOST", None)
if not host:
pytest.skip("LIBRARY_DESK_HOST not configured")
response = await _get_or_skip(f"{str(host).rstrip('/')}/health", "library-desk")
assert response.status_code == 200
@pytest.mark.contract
class TestRedisContract:
"""Boundary: Redis on the configured memory DB."""
async def test_roundtrip(self):
import redis.asyncio as redis
client = redis.Redis(
host=config.REDIS_HOST,
port=config.REDIS_PORT,
db=config.REDIS_MEMORY_DB,
socket_connect_timeout=3,
)
try:
await client.ping()
except Exception as e:
pytest.skip(f"redis unreachable: {e}")
try:
await client.set("contract-test-key", "ok", ex=30)
assert await client.get("contract-test-key") == b"ok"
await client.delete("contract-test-key")
finally:
await client.aclose()