tatlock/CHANGELOG.md

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.2.5] - 2025-12-07

### Added

#### Phase 2: The Steward (Two-Tier Architecture)
- **The Steward Agent**: First-tier LLM agent for request analysis and capability recommendation
  - Analyzes requests with full conversation context awareness
  - Recommends relevant household capabilities for each request
  - Detects missing capabilities and provides guidance
  - Estimates request complexity (simple/moderate/complex)
  - Uses same Ollama model as Tatlock for VRAM efficiency

- **Household Registry**: Centralized capability management system
  - `HouseholdRegistry` for registering capabilities and toolsets
  - `HouseholdCapability` executive summaries for coordination
  - `HouseholdMember` specifications with PydanticAI toolsets
  - Domain-based tool organization (e.g., `src/agents/tatlock_core/`)
  - Dynamic tool scoping per request

- **Request Preprocessing Pipeline**: Steward → Tatlock flow integration
  - `preprocess_request()` orchestrates Steward analysis
  - Creates scoped toolsets based on recommendations
  - Formats Steward notes for Butler (conversation context included)
  - Integrated with Responses API via `create_response_with_steward()`

- **Tool Usage Tracking**: Benchmarking and accuracy analysis
  - `ToolCallTracker` for monitoring recommended vs. actual tool usage
  - Tracks recommendation accuracy metrics
  - Records benchmarks to Redis for cross-session analysis
  - Supports precision/recall/F1 score calculation

- **Streaming Transparency**: Real-time Steward analysis visibility
  - Streams Steward's reasoning as reasoning summary deltas
  - Streams Tatlock's response as output text deltas
  - Full SSE support for Steward + Tatlock flow
  - Conversation context and missing capabilities visible in stream

- **Structured Logging**: Operation timing and metadata tracking
  - `structlog`-based JSON logging for machine parsing
  - Context managers for automatic operation timing
  - Metadata enrichment for debugging and analysis
  - Integrated with benchmark recording

- **Redis Benchmark Storage**: Performance metrics persistence
  - Cross-session benchmark storage with 30-day expiry
  - Time-series metrics for Steward analysis and tool calls
  - Queryable by operation, time range, and metadata
  - Support for recommendation accuracy tracking

- **Benchmark Analysis Tools**: Performance analysis CLI
  - `scripts/benchmark_analysis.py` for metric analysis
  - Steward performance statistics (latency, success rate, recommendations)
  - Tool recommendation accuracy analysis (precision, recall, F1)
  - Per-tool accuracy breakdown and duration statistics

- **End-to-End Test Suite**: Comprehensive API integration tests
  - 17 E2E tests making real HTTP requests to running server
  - Tests for Chat Completions, Responses API, and streaming endpoints
  - OpenAI API spec compliance verification (format validation)
  - Steward preprocessing integration verification
  - Error handling tests (404, 422 status codes)
  - Flexible assertions for LLM output variance
  - Tool usage indicators: 🧮 (calculator), 🔍 (search), 🕐 (datetime)
  - Full documentation in `tests/e2e/README.md`

#### Phase 1 Enhancements
- **Conversation history support**: Tatlock now remembers previous turns in multi-turn conversations
  - OpenAI-format messages converted to PydanticAI `ModelRequest`/`ModelResponse` objects
  - Full conversation context passed to agent via `message_history` parameter
  - Empty messages filtered to prevent Ollama errors
- **Tool call logging to reasoning output**: Users can see what tools are doing in real-time
  - `ToolCallTracker` dependency system for per-request tool usage logging
  - Web search queries appear with 🔍 emoji (e.g., "🔍 Searching for: 'Python 3.13'")
  - Calculator expressions appear with 🧮 emoji (e.g., "🧮 Calculating: sqrt(144) + 25")
  - Date/time operations appear with 🕐 emoji (e.g., "🕐 Calculating date offset: 2 weeks ago")
  - Tool usage visible in `<think>` tags in Open WebUI

### Changed
- **Architecture**: Two-tier request flow (Steward analysis → Tatlock execution)
- **Tool Organization**: Tatlock core tools reorganized into domain directory
- **Tool Scoping**: Tatlock runs with dynamically scoped toolsets per request
- **Responses API**: Integrated Steward preprocessing for all Tatlock requests
- **Streaming**: Enhanced to include Steward reasoning transparency
- Enhanced Tatlock agent with conversation memory capabilities
- All tools now log their usage via `RunContext` dependencies
- Improved debug logging for message history construction

### Fixed
- **Streaming text repetition**: Fixed text accumulation bug causing repetitive output in Open WebUI
  - Changed from accumulated text to delta mode (`stream_text(delta=True)`)
  - Implemented proper `run_with_scoped_tools_stream()` using PydanticAI's `run_stream()`
  - Replaced artificial word-by-word chunking with real LLM deltas
- **Broken tool execution in streaming**: Tools now execute properly in streaming mode
  - Previously showed raw JSON function calls instead of executed results
  - Now properly streams tool execution results
- **Invalid schema parameter**: Removed invalid `thinking` parameter from `ReasoningOutputItem`
- **Case sensitivity in model routing**: Model comparison now case-insensitive (`.lower()`)
- Conversation context now properly maintained across multiple turns
- Tool usage transparency - users can see exactly what queries/calculations are being performed
- Schema object handling in usage calculation (_calculate_usage reordered isinstance checks)

## [0.2.0] - 2025-12-06

### Added

#### PydanticAI Integration (Phase 1)
- Real Tatlock agent using PydanticAI with Ollama backend (mistral-nemo:latest)
- British butler personality with research-oriented mindset
- Lazy agent initialization to avoid connection issues in tests
- Streaming response integration with reasoning output
- Error handling for PydanticAI-specific exceptions

#### Permanent Tools (Phase 1)
- **Calculator tool** (`src/agents/tools.py`):
  - Safe mathematical expression evaluation using restricted namespace
  - Support for arithmetic, algebra, trigonometry, logarithms
  - Math functions: sqrt, sin, cos, tan, log, exp, etc.
  - Constants: pi, e
  - Integer result formatting (removes unnecessary decimals)
- **Date/Time toolkit**:
  - `get_current_datetime`: Current date/time in multiple formats
  - `calculate_time_offset`: Relative date calculations ("1 week ago", "2 months from now")
  - `time_difference`: Human-readable time differences between dates
- **Web Search tool**:
  - SearXNG integration for privacy-preserving web search
  - Automatic fallback from production to localhost in development
  - Formatted search results with titles, URLs, and snippets
  - Configurable result limits (max 10)

#### Tool Framework
- PydanticAI tool registration with `@agent.tool` decorator
- Tool descriptions visible to LLM for intelligent usage
- Async tool support for I/O operations
- Error handling with string-based error messages
- Tool usage guidelines in system prompt

#### Configuration
- SearXNG configuration in `src/core/config.py`:
  - `SEARXNG_HOST` with development fallback
  - `SEARXNG_TIMEOUT` setting
- Updated `.env.example` with SearXNG configuration
- Ollama configuration documentation

#### Testing
- 26 new tool tests (`tests/agents/test_tools.py`):
  - 7 calculator tests (arithmetic, functions, error handling)
  - 14 date/time tests (current time, offsets, differences)
  - 5 web search tests (mocked HTTP client)
- Updated registry tests for tools capability
- Total: 131 tests, 81.78% coverage (up from 95 tests, 78.95%)

#### Documentation
- Comprehensive README.md updates:
  - Tatlock agent capabilities and tool descriptions
  - Requirements section with Ollama and SearXNG setup
  - Configuration examples for external services
  - Tool usage examples and philosophy
  - Troubleshooting for Ollama and SearXNG
  - Updated test statistics
- AGENTS.md refactored for LLM development:
  - PydanticAI tool registration pattern
  - Tool implementation guidelines
  - Removed project status, focused on development instructions
- IMPLEMENTATION_ROADMAP.md updates:
  - Phase 1 marked as "MOSTLY COMPLETE"
  - Detailed completion status for each deliverable
  - Updated current state summary

### Changed
- Tatlock agent converted from mock to real PydanticAI implementation
- Tatlock capabilities updated: `tools: True`
- Streaming coordination now handles chunk-based delivery (50 chars) to preserve markdown
- Chat service streaming updated to preserve formatting
- System prompt enhanced with tool usage guidelines and research mindset
- Agent initialization changed to lazy pattern for better testability

### Fixed
- Text duplication bug in streaming responses (proper delta calculation)
- Markdown formatting preservation in streamed responses
- GeneratorExit errors from async context managers in generators
- PydanticAI API usage (`result.output` instead of `result.data`)

## [0.1.1] - 2025-12-06

### Added

#### Agent Interface (Phase 1)
- Abstract `AgentInterface` base class for model abstraction
- `LoremTesterAgent`: Full-featured mock agent with realistic behavior
  - Configurable reasoning effort levels (none, minimal, low, medium, high, xhigh)
  - Random tool/function call generation for testing
  - Error triggers: rate_limit, context_overflow, invalid_tool
  - Temperature-based response variation
- `TatlockAgent`: Placeholder for future PydanticAI integration
- `ModelRegistry`: Centralized model management and discovery
- 18 agent tests with comprehensive coverage

#### Responses API (Phases 2, 3, 6)
- OpenAI Responses API format with structured output (`/v1/responses`)
  - Reasoning items (thinking summaries with configurable effort)
  - Function call items (tool execution simulation)
  - Message items (assistant responses with output_text)
  - Streaming and non-streaming modes
- Real-time streaming with SSE-Starlette
- Conversation history management (Phase 3):
  - Hybrid client/server approach (client maintains state, server tracks)
  - Auto-generated deterministic conversation IDs from message hash
  - Configurable max turns with automatic trimming (default: 20)
  - Context window management with approximate token counting
  - Token usage statistics
  - Placeholder for future vector memory (Qdrant)
- Advanced features (Phase 6):
  - Parameter validation with Pydantic field validators
  - Temperature: 0.0-2.0 range enforcement
  - Reasoning effort: 6 levels validation
  - Max output tokens: positive integer enforcement
  - Stop sequences: up to 4, non-empty strings
  - Real-time stop sequence detection during streaming
  - Real-time max tokens enforcement with token counting
- 45 Responses API tests (router, error handling, history, advanced features)

#### Chat Completions Wrapper (Phase 5)
- OpenAI Chat Completions compatibility layer (`/v1/chat/completions`)
- Single source of truth architecture (wraps Responses API)
- Automatic reasoning generation
- Converts reasoning items to `<think>` tags for Open WebUI
- Pipeline prefix preservation
- System message support
- Enhanced error types (RateLimitError, ContextLengthError)
- 12 Chat Completions tests (router + streaming wrapper)

#### Application Infrastructure
- FastAPI application factory pattern
- CORS middleware with configurable origins
- Global exception handlers:
  - AppException handler for custom errors
  - RequestValidationError handler for Pydantic validation
  - General exception handler for unexpected errors
- Lifespan management for startup/shutdown
- OpenAPI schema with interactive documentation
- 16 main application tests

#### Testing Infrastructure
- Comprehensive test suite: 95 tests, 78.95% coverage (up from 62%)
- Async test support with pytest-asyncio
- Test fixtures for sync and async clients
- Integration tests for all API endpoints
- Streaming functionality tests
- Parameter validation tests
- Error handling tests
- Conversation history tests

#### Documentation
- Complete README.md rewrite with hybrid architecture
- Architecture diagrams and decision documentation
- AGENTS.md with technical implementation details
- API usage examples for all endpoints
- Conversation history guide
- Open WebUI integration instructions
- Troubleshooting section
- Implementation planning documents

### Changed
- Hybrid architecture with Responses API as primary endpoint
- Chat Completions now wraps Responses API (no duplicate logic)
- Enhanced error handling with OpenAI-compatible format
- Improved streaming with word-by-word delivery
- Better test organization with domain-based structure

### Security
- Minor version locking for all dependencies
- All packages CVE-checked (as of 2025-12-06)
- Environment variable protection via .gitignore
- No known vulnerabilities in dependency tree
- Input validation on all API endpoints

## [0.1.0] - 2025-12-06

### Added
- Project initialization
- Python 3.12.11 environment
- FastAPI 0.123.9 web framework
- PydanticAI 1.27.0 dependency (ready for future integration)
- Mock chat completions (lorem ipsum responses)
- Mock model listing (mistral-nemo:latest)
- Testing infrastructure (pytest, coverage, ruff, mypy)
- Configuration management with pydantic-settings
- CORS middleware
- Exception handlers (OpenAI-compatible error format)

[Unreleased]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v0.2.0...main
[0.2.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v0.1.1...v0.2.0
[0.1.1]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v0.1.0...v0.1.1
[0.1.0]: https://git.schweitz.net/jpmschweitzer/tatlock/releases/tag/v0.1.0