# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [2.1.0] - 2026-02-05

### Fixed

- **Streaming SSE compatibility with Open WebUI** - Switch from `exclude_none=True` to `exclude_unset=True` for SSE chunk serialization; `exclude_none` was too aggressive — it stripped `finish_reason: null` from intermediate chunks (which OpenAI includes), while `exclude_unset` correctly omits only fields never passed to the constructor (like `reasoning_content` on content-only chunks) while preserving explicitly-set `finish_reason: null`

### Changed

- **Project structure consolidation** - Moved documentation to `docs/`, consolidated all config into `pyproject.toml`, replaced `wakeup.sh`/`pytest.ini`/`requirements*.txt` with `Makefile` + `pyproject.toml`
- **CI test gate** - Unit tests now gate release and build jobs in Gitea Actions workflow
- **Build output organization** - Tool caches in `.cache/`, generated output (coverage, logs) in `build/`

## [2.0.5] - 2026-02-05

### Fixed

- **Streaming JSON compatibility** - Exclude null fields from streaming chunks using `exclude_none=True`; OpenAI's API omits null fields entirely, and including them (e.g., `content: null`, `reasoning_content: null`) caused parsing issues in Open WebUI

## [2.0.4] - 2026-02-05

### Fixed

- **Open WebUI streaming compatibility** - Replaced `sse_starlette` `EventSourceResponse` with plain `StreamingResponse` for chat completions; `sse_starlette` added `\r\n` line endings and extra SSE fields that Open WebUI couldn't parse

## [2.0.3] - 2026-02-05

### Fixed

- **Steward analysis leaking into responses** - Removed internal routing analysis (`DELEGATE: tatlock_core...`) from user-visible reasoning in both streaming and non-streaming paths

## [2.0.2] - 2026-02-05

### Fixed

- **tool_choice format incompatibility** - Removed `extra_body` tool_choice hack for Claude backend; PydanticAI handles tool_choice natively for Anthropic, preventing infinite tool call loops
- **CI trigger** - Changed workflow trigger from `release:published` to `push:tags:v[0-9]*`

## [2.0.1] - 2026-02-05

### Fixed

- **Expert agent registration failure** - `AnthropicModel` does not accept `api_key` directly; now passes it via `AnthropicProvider`

## [2.0.0] - 2026-02-05

### Added

- **Claude backend support (Claudification Phase 1)** - All agents now prefer Claude over Ollama
  - New `src/anthropic/` module with model selector and health check
  - `get_model()` factory returns Claude if available, Ollama as fallback
  - Startup health check caches Claude API availability
  - Configuration: `ANTHROPIC_API_KEY`, `ANTHROPIC_MODEL`, `PREFER_CLOUD_BACKEND`
  - 200k token context when using Claude backend

- **Steward dual-backend support** - Direct API calls to Claude or Ollama
  - `_call_claude()`: Anthropic Messages API path
  - `_call_ollama()`: Existing Ollama generate API path (preserved)
  - Automatic fallback: if Claude call fails mid-request, retries with Ollama

- **Claudification project tracking** - `PROJECT_CLAUDIFICATION.md` with Phase 1/2 roadmap

### Changed

- **All PydanticAI agents refactored to use `get_model()`**:
  - Tatlock (6 instantiation locations)
  - Librarian
  - Biographer
  - Housekeeper
- **`initialize_application()` is now async** - Supports async Claude health check at startup
- **Dependencies**: `pydantic-ai-slim[openai,anthropic]` replaces `pydantic-ai-slim[openai]`
- **Startup logging** now includes backend selection info (claude/ollama)
- **Agent creation logging** now includes backend and model info

### Removed

- Stale `tests/core/test_benchmarks.py` (benchmark system was removed in v1.10.0)

## [1.11.0] - 2025-12-30

### Added

- **Paperless document integration** - HybridRAG now includes indexed PDFs and scanned documents from Paperless-ngx
  - New `include_documents` parameter in `hybrid_search` tool
  - 📑 icon for document sources in search results
  - Librarian prompt updated with document awareness

- **Volatile cache integration** - HybridRAG now includes pre-fetched real-time data
  - New `include_volatile` parameter in `hybrid_search` tool
  - ⚡ icon for volatile sources in search results
  - Supports weather, forecast, news, stock, crypto, sun, air_quality namespaces
  - Librarian prompt updated with volatile cache awareness (user-configured items only)

- **Biographer routing in Steward** - Personal memory queries now correctly route to The Biographer
  - Added explicit routing rules for "where do I live", "what car do I drive", etc.
  - Added biographer delegation examples to Steward prompt
  - Location keywords ("live", "where", "home") now trigger profile pre-fetch

### Changed

- **LibraryDeskClient.hybrid_search** - Now passes full config including `document_limit`, `volatile_limit`, and enable flags
- **Steward guidelines** - Clarified that research queries about TOPICS go to Librarian, queries about USER go to Biographer

## [1.10.1] - 2025-12-23

### Fixed

- **Tatlock's excessive apologizing** - Strengthened personality prompt to prevent unnecessary apologies after successful Librarian delegations. Added explicit "do NOT apologize" instructions to both system prompt and synthesis prompt.

## [1.10.0] - 2025-12-22

### Added

#### Lightweight Request Tracing
- **JSON-based tracing system** for local development debugging
  - Captures full request flow through multi-agent architecture
  - `Trace` and `Span` dataclasses with automatic timing and nesting
  - ContextVar-based propagation for async-safe tracing
  - `trace_span` async context manager for clean instrumentation
  - Traces written to `logs/traces/{trace_id}.json`
  - Enabled via `DEBUG=true` environment variable
- **Trace Viewer UI** (`logs/traces/viewer.html`)
  - Standalone HTML viewer with timeline visualization
  - Filter by status, search by request text
  - Expandable span details with prompts and responses
- **Tracing REST API** (`/traces`)
  - `GET /traces` - Serve trace viewer UI
  - `GET /traces/list` - List available traces with filtering
  - `GET /traces/{trace_id}` - Retrieve specific trace JSON
  - Only available when `DEBUG=true`
- **Full pipeline instrumentation**
  - Router-level trace start/end with context management
  - Steward analysis spans in preprocessing
  - Tatlock orchestrate/synthesize spans
  - Expert delegation spans (librarian/biographer/housekeeper)
  - Tool-level spans extracted from PydanticAI messages

### Changed

- **Replaced Redis benchmarks with file-based tracing** - Simpler, more useful for debugging
- **Context management moved to service layer** - Router simplified, context set in response service
- **Server binds to all interfaces** - `wakeup.sh` now uses `0.0.0.0` for network access

### Removed

- **Redis benchmark system** (`src/core/benchmarks.py`)
  - `ENABLE_BENCHMARKS` config setting
  - `REDIS_BENCHMARK_DB` config setting
  - `redis_url` property (kept `redis_memory_url`)
  - Benchmark recording in Steward service and tool tracking

### Fixed

- **Librarian fabrication prevention** - Added explicit instructions to never invent data when tools fail or sources are unavailable

## [1.9.0] - 2025-12-18

### Changed

- **Housekeeper prompt optimization** - Rewrote system prompt for Mistral-Nemo function calling with negative constraints, step-by-step process, and explicit entity ID format guidance
- **Housekeeper temperature setting** - Set temperature to 0.1 for deterministic tool calling behavior
- **Device list room group priority** - Room groups now appear first in `list_devices` output with `[ROOM GROUP]` marker to address positional bias
- **Tool docstring improvements** - Updated turn_on/turn_off/toggle with explicit `entity_id=` parameter examples

### Added

- **Housekeeper optimization findings** - Added `docs/housekeeper-optimization-findings.md` documenting the experiment journey from 0% to 100% success rate
- **Housekeeper test script** - Added `scripts/test_housekeeper.sh` for room group detection regression testing

## [1.8.6] - 2025-12-17

### Fixed

- **Housekeeper API paths** - Updated all client endpoints to use `/housekeeping/` prefix to match core-api routes
- **Housekeeper entity hallucination** - Improved system prompt with critical rule requiring `list_devices()` before any control action to prevent guessing entity IDs

### Added

- **Housekeeping API spec** - Added `docs/housekeeping-api-spec.md` documenting the core-api home automation interface

## [1.8.5] - 2025-12-16

### Fixed

- **Redis benchmark boolean storage** - Convert booleans to strings for Redis `hset` (Redis doesn't accept bool type directly)
- **Tool tracking capability matching** - `delegate_to_librarian` now correctly recognized as using "librarian" capability when checking Steward recommendations
- **E2E test fixture scope** - Fixed pytest-asyncio ScopeMismatch error by using `loop_scope="module"` for module-scoped async fixtures

## [1.8.4] - 2025-12-16

### Fixed

- **Remove `<think>` wrappers from think messages** - Messages in `reasoning_content` should be plain text
- Removed `<think>` wrappers from delegation.py household think messages
- Removed `<think>` wrappers from orchestration.py status messages
- Think messages now appear cleanly in Open WebUI's reasoning block

## [1.8.3] - 2025-12-16

### Fixed

- **Open WebUI streaming rendering** - Use `reasoning_content` field for thinking (DeepSeek R1 format) instead of `<think>` tags in `content`
- Open WebUI now renders thinking as proper collapsible blocks instead of broken HTML

## [1.8.2] - 2025-12-16

### Fixed

- **HybridRAG keywords schema mismatch** - library-desk now returns `keywords` as dict with `core_keywords`, client now handles both formats

## [1.8.1] - 2025-12-16

### Fixed

#### Ollama Message Sanitization
- **Fixed `invalid message content type: <nil>` error** from Ollama
- Created custom `TatlockOllamaProvider` that sanitizes messages before sending to Ollama
- Ollama rejects assistant messages with `content: null` (tool-only messages from PydanticAI)
- Provider converts `null` content to empty string `""` for compatibility
- Updated all agents (Librarian, Biographer, Housekeeper, Tatlock) to use sanitized provider
- Added `src/ollama/provider.py` with reusable provider pattern

#### Streaming Think Message Accumulation
- **Fixed repeating think messages in frontend** (e.g., 10x "The Librarian has compiled...")
- Frontend was accumulating `ReasoningSummaryDelta` events expecting concatenation
- Added `ReasoningSummaryDone()` signal after each think message to indicate completion
- Each think slug is now treated as a complete message, not a continuation

## [1.8.0] - 2025-12-15

### Fixed

#### Steward Routing for Web Search
- Updated Steward guidelines to route web searches, weather, news → Librarian with `search_web`
- Added URL/article reading → Librarian with `read_url` to routing guidelines
- Added examples showing `search_web` and `read_url` tool usage

#### Librarian Agent Tool Registration
- Registered `search_web`, `read_url`, `read_urls_batch` tools with the Librarian PydanticAI agent
- Updated Librarian system prompt with Web Search & Content Extraction section
- Fixed tool count in agent logger (11 → 14 tools)

#### Query Enrichment Integration
- Fixed enriched query (with location/timezone context) not being passed to delegations
- Response service now uses `enriched_query` from Steward recommendation for all delegations
- Weather queries now automatically include user's stored location

#### Action Type Detection
- Added "read", "fetch", "url", "http" keywords to RESEARCH action type for Librarian
- Ensures proper think messages for URL reading tasks

## [1.7.0] - 2025-12-15

### Added

#### Web Search Migration to Librarian
- **`search_web()`** tool in Librarian for web search via library-desk `/rag/search` endpoint
- **`read_url()`** tool for single URL content extraction via Trafilatura
- **`read_urls_batch()`** tool for parallel batch URL extraction (max 20 URLs)
- `WebSearchResult`, `WebSearchResponse` models in LibraryDeskClient
- `ContentExtractionResult`, `BatchExtractionResponse` models for content extraction
- `search_web()`, `extract_content()`, `extract_content_batch()` methods in LibraryDeskClient
- Comprehensive unit tests for new Librarian tools (`tests/agents/librarian/test_tools.py`)

### Changed

- Librarian capability updated with web search domains: "web", "url", "internet"
- Tatlock system prompt now delegates web search to Librarian
- `tatlock_core` capability reduced to computation/datetime only (no longer requires network)

### Removed

- `search_web` function from `src/agents/tatlock_core/tools.py`
- `web_search_tool` from `tatlock_core_tools` list
- `search_web` from legacy `src/agents/tools.py`
- Search tests from `tests/agents/test_tools.py` (moved to Librarian tests)

## [1.6.0] - 2025-12-15

### Added

#### Two-Phase Tatlock Execution
- **Phase 1: Orchestration** - Executes tool calls and expert delegations, returns structured results
- **Phase 2: Synthesis** - Synthesizes butler-toned response from gathered results
- `orchestrate_tool_calls()` method in TatlockAgent for coordination phase
- `synthesize_from_results()` method in TatlockAgent for synthesis phase
- Guarantees butler personality in all responses by separating coordination from response generation

#### Automatic Think Slugs
- **Deterministic butler-perspective messages** during expert delegation (no LLM involved)
- `ActionType` enum: RETRIEVE, RESEARCH, CREATE, CONTROL, RECORD
- `HOUSEHOLD_THINK_MESSAGES` mapping with butler-perspective messages for all experts:
  - Librarian: "Allow me to consult the archives, sir." / "I'm having the Librarian prepare a new entry."
  - Biographer: "Let me consult the household records." / "I've asked the Biographer to take note, sir."
  - Housekeeper: "I'm instructing the household staff now, sir." / "Allow me to inquire with the household staff."
- `_detect_action_type()` function for keyword-based action detection
- `get_think_message()` helper for retrieving appropriate messages
- Streaming delegation wrappers: `stream_delegate_to_librarian()`, `stream_delegate_to_biographer()`, `stream_delegate_to_housekeeper()`
- `STREAMING_DELEGATION_WRAPPERS` mapping in delegation.py
- `get_streaming_delegation_tools()` method in HouseholdRegistry

#### Steward Query Enrichment
- **Auto-fill user context** (location, timezone) when not specified in query
- `_build_enriched_query()` function in steward service
- Regex word boundary matching for accurate location detection (avoids false positives)
- `enriched_query` field added to `StewardRecommendation` schema
- Automatic enrichment for weather queries (location), time queries (timezone), temperature preferences

#### Documentation
- **ORCHESTRATION_SCENARIOS.md** completely rewritten with:
  - Mermaid flow diagrams for two-phase execution
  - 4 new Housekeeper scenarios (light control, device status, parallel delegation)
  - Biographer memory recording scenario
  - Complete think slug reference tables
  - Action type detection tables
  - Updated architecture mindmap
- **TESTING_IMPROVEMENTS.md** - LLM testing best practices for future implementation

### Changed

- `create_response_with_steward()` now uses two-phase execution
- `_direct_delegation()` routes through synthesis phase for consistent butler tone
- `_execute_single_delegation()` now supports housekeeper
- Streaming response handler integrated with think slug system
- All 326 unit tests passing

## [1.5.0] - 2025-12-15

### Added

#### The Housekeeper Agent
- **New home automation expert agent** following the Librarian pattern
- `CoreAPIClient` for communicating with core-api service (Home Assistant wrapper)
- 13 tools for home automation:
  - Discovery: `list_areas`, `list_devices`, `get_device_state`
  - Control: `turn_on`, `turn_off`, `toggle`
  - Scenes: `list_scenes`, `activate_scene`
  - Scripts: `list_scripts`, `run_script`
  - Automations: `list_automations`, `toggle_automation`
  - History: `get_history`
- PydanticAI agent with system prompt for home automation tasks
- `HouseholdCapability` registration with domains: lights, switches, automation, home, smart home, scene, script, device, climate, fan, cover, blinds
- `delegate_to_housekeeper()` delegation wrapper
- Config settings: `CORE_API_HOST`, `CORE_API_KEY`, `CORE_API_TIMEOUT`

#### Development Port Change
- **Dev server port changed from 8123 to 8777** to avoid conflict with Home Assistant default port
- Updated `wakeup.sh`, E2E tests, and documentation

### Changed

- All unit tests pass (421 passed, 5 xfailed)
- Housekeeper registered on startup alongside Librarian and Biographer

## [1.4.0] - 2025-12-14

### Added

#### Environment-Aware Configuration
- **Auto-selected logging level**: DEBUG for development, WARNING for production
- **Auto-selected default user**: `llm_tester` for development (isolated test scope), `jpmschweitzer` for production
- Properties `effective_log_level` and `effective_default_user` in config
- User context logging at request entry with INFO level

#### Direct Delegation Bypass
- **Pure memory/librarian requests bypass Tatlock**: When Steward recommends only biographer/librarian, skip Tatlock LLM call
- `_direct_delegation()` function for immediate expert agent execution
- Reduces latency for memory-only requests

#### Text-Based Delegation Fallback
- **Parse text delegation patterns**: Handle LLM outputs like `[DELEGATE:biographer] task="..."`
- Multiple pattern support for delegation parsing
- Sequential and parallel execution with `[PARALLEL]` prefix

#### Comprehensive E2E Test Suite
- **22 new orchestration tests** in `tests/e2e/test_orchestration_e2e.py`
- `QdrantVerifier` helper class for data verification
- `assert_llm_behavior()` for flexible LLM output pattern matching
- Test classes covering:
  - Memory storage and recall
  - Steward delegation
  - Direct delegation bypass
  - User context isolation (llm_tester vs production)
  - Data verification in Qdrant
  - Integration health checks
  - Orchestration scenarios (weather, calculator, wiki, multi-expert)
  - Error handling
  - Evaluation reports
- Updated `tests/e2e/README.md` with comprehensive documentation

### Fixed

- **Unit test mocks**: Updated Steward streaming tests to mock `run_with_scoped_tools_stream` (async generator)
- **Temporal context in tests**: Tests now account for `_inject_temporal_context()` appending timestamps
- **LLM non-determinism**: Integration tests use `pytest.xfail()` for LLM-dependent assertions
- **Streaming test timeouts**: Increased timeouts (60-90s) for LLM processing time

### Changed

- All unit tests now pass (380 passed, 5 xfailed for LLM non-determinism)
- E2E tests use `llm_tester` user for isolation from production data

## [1.3.3] - 2025-12-14

### Fixed

- **Memory**: Fix Qdrant point IDs - use UUID5 instead of arbitrary strings

## [1.3.2] - 2025-12-14

### Fixed

- **Memory**: Fix biographer tool type hints for Ollama compatibility (remove `| None` union types)

## [1.3.1] - 2025-12-14

### Fixed

- **Memory**: Add biographer to delegation wrappers (was returning raw tools causing Ollama error)
- **Config**: Add Qdrant host/port to .env.example

## [1.3.0] - 2025-12-14

### Fixed

- **Memory**: Update Qdrant client to use `query_points` API (qdrant-client >= 1.10)

### Changed

- **Config**: Rename `REDIS_DB` to `REDIS_BENCHMARK_DB` for clarity
- **Config**: Update Redis defaults to match stack allocation (benchmark=6, memory=1)

## [1.2.5] - 2025-12-14

### Fixed

- **Dependencies**: Add missing `pydantic-settings` (not included in pydantic-ai-slim)

## [1.2.4] - 2025-12-14

### Added

- **CI**: Trigger Watchtower update after successful image push

## [1.2.3] - 2025-12-14

### Fixed

- **CI**: Upgrade to build-push-action@v6, disable provenance and sbom for Gitea registry

## [1.2.2] - 2025-12-13

### Fixed

- **CI**: Add `provenance: false` to docker/build-push-action to fix Gitea registry push

## [1.2.1] - 2025-12-13

### Changed

- **Dependency slimming**: Switched from `pydantic-ai` to `pydantic-ai-slim[openai]`
  - Removes unused LLM provider SDKs (anthropic, boto3, cohere, google-genai, groq, huggingface)
  - Production packages: 53 (down from ~158)
  - Production footprint: 178MB
  - Tatlock uses Ollama via OpenAI-compatible API, so only `openai` extra is needed
  - See `DEPENDENCY_SLIM.md` for rollback instructions

## [1.2.0] - 2025-12-13

### Added

#### Phase F: Memory System (The Biographer)

- **Memory Infrastructure** (Phase F.1):
  - `src/core/context.py`: ContextVar-based request context for async-safe user/conversation tracking
    - `get_user()`, `get_conversation_id()` helpers
    - `RequestContext` manager for clean setup/teardown
  - `src/core/multi_tenancy.py`: User ID sanitization and collection naming
    - Per-user collection pattern: `memories_{user}`
    - Redis key patterns: `session:{user}:{conv}`, `entities:{user}:{conv}`
  - `src/core/embeddings.py`: Ollama embedding client
    - nomic-embed-text model (768 dimensions)
    - `embed()`, `embed_batch()`, `health_check()` methods
  - `src/core/qdrant.py`: Qdrant vector database client
    - `ensure_collection()`, `upsert_memory()`, `search_memories()`, `delete_memory()`
    - Type-based filtering for memory queries
  - `src/core/memory_cache.py`: Redis session memory cache
    - Session context with 24h TTL (db=2, separate from benchmarks)
    - Recent entities tracking per conversation

- **Memory Service** (Phase F.2a):
  - `src/core/memory_service.py`: Direct access layer for fast, LLM-free memory lookups
    - Profile methods: `get_profile()`, `set_profile()`
    - Preference methods: `get_preference()`, `set_preference()`, `get_all_preferences()`
    - Fact methods: `store_fact()`, `get_fact()`
    - Session context: `get_session_context()`, `set_session_context()`, `update_session_context()`
    - Steward integration: `prefetch_context()` for request preprocessing

- **The Biographer Agent** (Phase F.2b):
  - `src/agents/biographer/`: Household memory keeper agent
    - PydanticAI agent with discreet chronicler personality
    - System prompt emphasizes privacy and accurate recall
  - **Biographer Tools** (`src/agents/biographer/tools.py`):
    - `recall_semantic`: Semantic search for memories by meaning
    - `list_memories`: Browse stored memories by type
    - `store_insight`: Record new facts from conversation
    - `update_profile`: Update core profile fields (name, location, timezone)
    - `update_preference`: Update user preferences (units, theme)
    - `forget_memory`: Remove specific memories
  - **Capability Registration**:
    - `BIOGRAPHER_CAPABILITY` with context domain
    - Automatic registration on startup
    - Low cost (vector search, minimal LLM)

- **Delegation Wrapper**:
  - `delegate_to_biographer()` in `src/agents/delegation.py`
  - Async delegation with error handling

- **Steward Memory Integration**:
  - Memory context pre-fetch during request analysis
  - Profile and preferences included in Steward's note to Butler
  - Keyword-based context determination (weather → location, time → timezone)

- **Configuration**:
  - `QDRANT_HOST`, `QDRANT_PORT`, `QDRANT_EMBEDDING_DIM` (768)
  - `OLLAMA_EMBEDDING_MODEL` (nomic-embed-text)
  - `REDIS_MEMORY_DB` (2), `REDIS_MEMORY_TTL_HOURS` (24)

- **Test Suite**:
  - 34 new tests for memory system
  - Biographer capability tests (15 tests)
  - Memory service tests (19 tests)

- **OpenAI Standard `user` Field**:
  - Added `user` field to `ResponseRequest` schema
  - Request context set at API entry point
  - Propagates through async calls via ContextVar

### Changed
- Application startup now registers The Biographer with Household Registry
- Steward analysis includes memory context pre-fetch
- Librarian client methods now use `get_user()` from context (12 methods updated)
- Request router sets user/conversation context at entry

## [1.1.0] - 2025-12-11

### Added

#### Phase 3: Butler Orchestration (Multi-Agent Coordination)
- **The Librarian Agent**: Expert agent for research and knowledge management
  - PydanticAI agent with specialized research assistant personality
  - Connects to library-desk API for HybridRAG capabilities
  - System prompt emphasizes fetching wiki pages before summarizing
  - Streaming support via `run_librarian_stream()`

- **Library-Desk API Client** (`src/agents/librarian/client.py`):
  - Async HTTP client with httpx for library-desk API integration
  - HybridRAG search (vector + graph + web search)
  - Wiki operations (search, get, list, create, update pages)
  - Smart page creation with HybridRAG research (`POST /wiki/pages/smart-create`)
  - Semantic vector search
  - Knowledge graph queries (Cypher execution)
  - Dossier (tag collection) browsing
  - Health check endpoint

- **Librarian Tools** (`src/agents/librarian/tools.py`):
  - Research tools:
    - `hybrid_search`: Combined vector, graph, and web search
    - `search_wiki`: Full-text wiki page search
    - `get_wiki_page`: Fetch full wiki page content by ID
    - `semantic_search`: Vector similarity search
    - `list_dossiers`: Browse knowledge collections
    - `get_dossier_pages`: Get pages in a dossier
    - `explore_knowledge_graph`: Entity and relationship discovery
    - `find_related_entities`: Find connected concepts
  - Write tools:
    - `smart_create_wiki_page`: Create page with automatic HybridRAG research (PREFERRED for topic-based creation)
    - `create_wiki_page`: Create page with user-provided content
    - `update_wiki_page`: Update existing page (partial updates supported)

- **Agent Communication Protocol** (`src/agents/protocol.py`):
  - `AgentRequest`: Standardized task request with context and constraints
  - `AgentResponse`: Response with result, reasoning, tool calls, confidence
  - `DelegationIntent`: Routing intent with target agent and reason
  - `CoordinationResult`: Aggregated multi-agent results
  - `DelegationReason` enum: domain expertise, tool access, resource efficiency, user preference
  - Error types: `AgentError`, `AgentTimeoutError`, `AgentUnavailableError`

- **Coordination Engine** (`src/agents/coordination.py`):
  - `CoordinationEngine`: Multi-agent task orchestration
  - Routing tasks to appropriate expert agents
  - Sequential and parallel execution support
  - Result aggregation from multiple agents
  - Graceful error handling and degradation
  - Streaming delegation support
  - Convenience functions: `delegate_to_librarian()`, `delegate_to_librarian_stream()`

- **Librarian Capability Registration**:
  - `LIBRARIAN_CAPABILITY` definition with research domains
  - Automatic registration on application startup
  - Integration with Household Registry

- **Configuration**:
  - `LIBRARY_DESK_HOST`: Library-desk API URL (default: `http://localhost:8089`)
  - `LIBRARY_DESK_API_KEY`: Optional API key for authentication
  - `LIBRARY_DESK_TIMEOUT`: Request timeout in seconds (default: 60)

- **Test Suite**:
  - 78 new tests for Phase 3 components
  - Protocol model tests (requests, responses, intents, errors)
  - Coordination engine tests (delegation, streaming, multi-agent)
  - Library-desk client tests (all endpoints with mocked HTTP)
  - Wiki write operation tests (update, smart-create)
  - Capability registration tests

### Changed
- Application startup now registers The Librarian with Household Registry
- Configuration expanded to support library-desk API integration
- **Version loading**: APP_VERSION now dynamically loaded from pyproject.toml

## [1.0.0a] - 2025-12-11

### Added
- **CI/CD Pipeline**: Release-triggered automated builds
  - Dockerfile for containerized deployment (Python 3.12-slim, port 8000)
  - Gitea Actions workflow triggered on release publish
  - Builds and pushes to git.schweitz.net registry with latest and version tags
  - Watchtower integration for automatic container updates
- **Portainer Stack**: Production deployment configuration
  - Connects to docker-dataplane network for service discovery
  - Integration with ollama, searxng, and redis-shared services
  - Health check endpoint monitoring
  - Resource limits (1 CPU, 1GB memory)

### Changed
- Version bump to 1.0.0 marking production-ready release

## [0.2.5] - 2025-12-07

### Added

#### Phase 2: The Steward (Two-Tier Architecture)
- **The Steward Agent**: First-tier LLM agent for request analysis and capability recommendation
  - Analyzes requests with full conversation context awareness
  - Recommends relevant household capabilities for each request
  - Detects missing capabilities and provides guidance
  - Estimates request complexity (simple/moderate/complex)
  - Uses same Ollama model as Tatlock for VRAM efficiency

- **Household Registry**: Centralized capability management system
  - `HouseholdRegistry` for registering capabilities and toolsets
  - `HouseholdCapability` executive summaries for coordination
  - `HouseholdMember` specifications with PydanticAI toolsets
  - Domain-based tool organization (e.g., `src/agents/tatlock_core/`)
  - Dynamic tool scoping per request

- **Request Preprocessing Pipeline**: Steward → Tatlock flow integration
  - `preprocess_request()` orchestrates Steward analysis
  - Creates scoped toolsets based on recommendations
  - Formats Steward notes for Butler (conversation context included)
  - Integrated with Responses API via `create_response_with_steward()`

- **Tool Usage Tracking**: Benchmarking and accuracy analysis
  - `ToolCallTracker` for monitoring recommended vs. actual tool usage
  - Tracks recommendation accuracy metrics
  - Records benchmarks to Redis for cross-session analysis
  - Supports precision/recall/F1 score calculation

- **Streaming Transparency**: Real-time Steward analysis visibility
  - Streams Steward's reasoning as reasoning summary deltas
  - Streams Tatlock's response as output text deltas
  - Full SSE support for Steward + Tatlock flow
  - Conversation context and missing capabilities visible in stream

- **Structured Logging**: Operation timing and metadata tracking
  - `structlog`-based JSON logging for machine parsing
  - Context managers for automatic operation timing
  - Metadata enrichment for debugging and analysis
  - Integrated with benchmark recording

- **Redis Benchmark Storage**: Performance metrics persistence
  - Cross-session benchmark storage with 30-day expiry
  - Time-series metrics for Steward analysis and tool calls
  - Queryable by operation, time range, and metadata
  - Support for recommendation accuracy tracking

- **Benchmark Analysis Tools**: Performance analysis CLI
  - `scripts/benchmark_analysis.py` for metric analysis
  - Steward performance statistics (latency, success rate, recommendations)
  - Tool recommendation accuracy analysis (precision, recall, F1)
  - Per-tool accuracy breakdown and duration statistics

- **End-to-End Test Suite**: Comprehensive API integration tests
  - 17 E2E tests making real HTTP requests to running server
  - Tests for Chat Completions, Responses API, and streaming endpoints
  - OpenAI API spec compliance verification (format validation)
  - Steward preprocessing integration verification
  - Error handling tests (404, 422 status codes)
  - Flexible assertions for LLM output variance
  - Tool usage indicators: 🧮 (calculator), 🔍 (search), 🕐 (datetime)
  - Full documentation in `tests/e2e/README.md`

#### Phase 1 Enhancements
- **Conversation history support**: Tatlock now remembers previous turns in multi-turn conversations
  - OpenAI-format messages converted to PydanticAI `ModelRequest`/`ModelResponse` objects
  - Full conversation context passed to agent via `message_history` parameter
  - Empty messages filtered to prevent Ollama errors
- **Tool call logging to reasoning output**: Users can see what tools are doing in real-time
  - `ToolCallTracker` dependency system for per-request tool usage logging
  - Web search queries appear with 🔍 emoji (e.g., "🔍 Searching for: 'Python 3.13'")
  - Calculator expressions appear with 🧮 emoji (e.g., "🧮 Calculating: sqrt(144) + 25")
  - Date/time operations appear with 🕐 emoji (e.g., "🕐 Calculating date offset: 2 weeks ago")
  - Tool usage visible in `<think>` tags in Open WebUI

### Changed
- **Architecture**: Two-tier request flow (Steward analysis → Tatlock execution)
- **Tool Organization**: Tatlock core tools reorganized into domain directory
- **Tool Scoping**: Tatlock runs with dynamically scoped toolsets per request
- **Responses API**: Integrated Steward preprocessing for all Tatlock requests
- **Streaming**: Enhanced to include Steward reasoning transparency
- Enhanced Tatlock agent with conversation memory capabilities
- All tools now log their usage via `RunContext` dependencies
- Improved debug logging for message history construction

### Fixed
- **Streaming text repetition**: Fixed text accumulation bug causing repetitive output in Open WebUI
  - Changed from accumulated text to delta mode (`stream_text(delta=True)`)
  - Implemented proper `run_with_scoped_tools_stream()` using PydanticAI's `run_stream()`
  - Replaced artificial word-by-word chunking with real LLM deltas
- **Broken tool execution in streaming**: Tools now execute properly in streaming mode
  - Previously showed raw JSON function calls instead of executed results
  - Now properly streams tool execution results
- **Invalid schema parameter**: Removed invalid `thinking` parameter from `ReasoningOutputItem`
- **Case sensitivity in model routing**: Model comparison now case-insensitive (`.lower()`)
- Conversation context now properly maintained across multiple turns
- Tool usage transparency - users can see exactly what queries/calculations are being performed
- Schema object handling in usage calculation (_calculate_usage reordered isinstance checks)

## [0.2.0] - 2025-12-06

### Added

#### PydanticAI Integration (Phase 1)
- Real Tatlock agent using PydanticAI with Ollama backend (mistral-nemo:latest)
- British butler personality with research-oriented mindset
- Lazy agent initialization to avoid connection issues in tests
- Streaming response integration with reasoning output
- Error handling for PydanticAI-specific exceptions

#### Permanent Tools (Phase 1)
- **Calculator tool** (`src/agents/tools.py`):
  - Safe mathematical expression evaluation using restricted namespace
  - Support for arithmetic, algebra, trigonometry, logarithms
  - Math functions: sqrt, sin, cos, tan, log, exp, etc.
  - Constants: pi, e
  - Integer result formatting (removes unnecessary decimals)
- **Date/Time toolkit**:
  - `get_current_datetime`: Current date/time in multiple formats
  - `calculate_time_offset`: Relative date calculations ("1 week ago", "2 months from now")
  - `time_difference`: Human-readable time differences between dates
- **Web Search tool**:
  - SearXNG integration for privacy-preserving web search
  - Automatic fallback from production to localhost in development
  - Formatted search results with titles, URLs, and snippets
  - Configurable result limits (max 10)

#### Tool Framework
- PydanticAI tool registration with `@agent.tool` decorator
- Tool descriptions visible to LLM for intelligent usage
- Async tool support for I/O operations
- Error handling with string-based error messages
- Tool usage guidelines in system prompt

#### Configuration
- SearXNG configuration in `src/core/config.py`:
  - `SEARXNG_HOST` with development fallback
  - `SEARXNG_TIMEOUT` setting
- Updated `.env.example` with SearXNG configuration
- Ollama configuration documentation

#### Testing
- 26 new tool tests (`tests/agents/test_tools.py`):
  - 7 calculator tests (arithmetic, functions, error handling)
  - 14 date/time tests (current time, offsets, differences)
  - 5 web search tests (mocked HTTP client)
- Updated registry tests for tools capability
- Total: 131 tests, 81.78% coverage (up from 95 tests, 78.95%)

#### Documentation
- Comprehensive README.md updates:
  - Tatlock agent capabilities and tool descriptions
  - Requirements section with Ollama and SearXNG setup
  - Configuration examples for external services
  - Tool usage examples and philosophy
  - Troubleshooting for Ollama and SearXNG
  - Updated test statistics
- AGENTS.md refactored for LLM development:
  - PydanticAI tool registration pattern
  - Tool implementation guidelines
  - Removed project status, focused on development instructions
- IMPLEMENTATION_ROADMAP.md updates:
  - Phase 1 marked as "MOSTLY COMPLETE"
  - Detailed completion status for each deliverable
  - Updated current state summary

### Changed
- Tatlock agent converted from mock to real PydanticAI implementation
- Tatlock capabilities updated: `tools: True`
- Streaming coordination now handles chunk-based delivery (50 chars) to preserve markdown
- Chat service streaming updated to preserve formatting
- System prompt enhanced with tool usage guidelines and research mindset
- Agent initialization changed to lazy pattern for better testability

### Fixed
- Text duplication bug in streaming responses (proper delta calculation)
- Markdown formatting preservation in streamed responses
- GeneratorExit errors from async context managers in generators
- PydanticAI API usage (`result.output` instead of `result.data`)

## [0.1.1] - 2025-12-06

### Added

#### Agent Interface (Phase 1)
- Abstract `AgentInterface` base class for model abstraction
- `LoremTesterAgent`: Full-featured mock agent with realistic behavior
  - Configurable reasoning effort levels (none, minimal, low, medium, high, xhigh)
  - Random tool/function call generation for testing
  - Error triggers: rate_limit, context_overflow, invalid_tool
  - Temperature-based response variation
- `TatlockAgent`: Placeholder for future PydanticAI integration
- `ModelRegistry`: Centralized model management and discovery
- 18 agent tests with comprehensive coverage

#### Responses API (Phases 2, 3, 6)
- OpenAI Responses API format with structured output (`/v1/responses`)
  - Reasoning items (thinking summaries with configurable effort)
  - Function call items (tool execution simulation)
  - Message items (assistant responses with output_text)
  - Streaming and non-streaming modes
- Real-time streaming with SSE-Starlette
- Conversation history management (Phase 3):
  - Hybrid client/server approach (client maintains state, server tracks)
  - Auto-generated deterministic conversation IDs from message hash
  - Configurable max turns with automatic trimming (default: 20)
  - Context window management with approximate token counting
  - Token usage statistics
  - Placeholder for future vector memory (Qdrant)
- Advanced features (Phase 6):
  - Parameter validation with Pydantic field validators
  - Temperature: 0.0-2.0 range enforcement
  - Reasoning effort: 6 levels validation
  - Max output tokens: positive integer enforcement
  - Stop sequences: up to 4, non-empty strings
  - Real-time stop sequence detection during streaming
  - Real-time max tokens enforcement with token counting
- 45 Responses API tests (router, error handling, history, advanced features)

#### Chat Completions Wrapper (Phase 5)
- OpenAI Chat Completions compatibility layer (`/v1/chat/completions`)
- Single source of truth architecture (wraps Responses API)
- Automatic reasoning generation
- Converts reasoning items to `<think>` tags for Open WebUI
- Pipeline prefix preservation
- System message support
- Enhanced error types (RateLimitError, ContextLengthError)
- 12 Chat Completions tests (router + streaming wrapper)

#### Application Infrastructure
- FastAPI application factory pattern
- CORS middleware with configurable origins
- Global exception handlers:
  - AppException handler for custom errors
  - RequestValidationError handler for Pydantic validation
  - General exception handler for unexpected errors
- Lifespan management for startup/shutdown
- OpenAPI schema with interactive documentation
- 16 main application tests

#### Testing Infrastructure
- Comprehensive test suite: 95 tests, 78.95% coverage (up from 62%)
- Async test support with pytest-asyncio
- Test fixtures for sync and async clients
- Integration tests for all API endpoints
- Streaming functionality tests
- Parameter validation tests
- Error handling tests
- Conversation history tests

#### Documentation
- Complete README.md rewrite with hybrid architecture
- Architecture diagrams and decision documentation
- AGENTS.md with technical implementation details
- API usage examples for all endpoints
- Conversation history guide
- Open WebUI integration instructions
- Troubleshooting section
- Implementation planning documents

### Changed
- Hybrid architecture with Responses API as primary endpoint
- Chat Completions now wraps Responses API (no duplicate logic)
- Enhanced error handling with OpenAI-compatible format
- Improved streaming with word-by-word delivery
- Better test organization with domain-based structure

### Security
- Minor version locking for all dependencies
- All packages CVE-checked (as of 2025-12-06)
- Environment variable protection via .gitignore
- No known vulnerabilities in dependency tree
- Input validation on all API endpoints

## [0.1.0] - 2025-12-06

### Added
- Project initialization
- Python 3.12.11 environment
- FastAPI 0.123.9 web framework
- PydanticAI 1.27.0 dependency (ready for future integration)
- Mock chat completions (lorem ipsum responses)
- Mock model listing (mistral-nemo:latest)
- Testing infrastructure (pytest, coverage, ruff, mypy)
- Configuration management with pydantic-settings
- CORS middleware
- Exception handlers (OpenAI-compatible error format)

[Unreleased]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v2.1.0...main
[2.1.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v2.0.5...v2.1.0
[2.0.5]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v2.0.0...v2.0.5
[2.0.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.11.0...v2.0.0
[1.11.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.10.0...v1.11.0
[1.10.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.9.0...v1.10.0
[1.9.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.8.6...v1.9.0
[1.8.6]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.8.5...v1.8.6
[1.8.5]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.8.4...v1.8.5
[1.8.4]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.8.3...v1.8.4
[1.8.3]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.8.2...v1.8.3
[1.8.2]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.8.1...v1.8.2
[1.8.1]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.8.0...v1.8.1
[1.8.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.7.0...v1.8.0
[1.7.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.6.0...v1.7.0
[1.6.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.5.0...v1.6.0
[1.5.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.4.0...v1.5.0
[1.4.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.3...v1.4.0
[1.3.3]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.2...v1.3.3
[1.3.2]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.1...v1.3.2
[1.3.1]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.0...v1.3.1
[1.3.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.2.5...v1.3.0
[1.2.5]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.2.4...v1.2.5
[1.2.4]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.2.3...v1.2.4
[1.2.3]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.2.2...v1.2.3
[1.2.2]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.2.1...v1.2.2
[1.2.1]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.2.0...v1.2.1
[1.2.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.1.0...v1.2.0
[1.1.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.0.0a...v1.1.0
[1.0.0a]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v0.2.5...v1.0.0a
[0.2.5]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v0.2.0...v0.2.5
[0.2.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v0.1.1...v0.2.0
[0.1.1]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v0.1.0...v0.1.1
[0.1.0]: https://git.schweitz.net/jpmschweitzer/tatlock/releases/tag/v0.1.0