docs: update changelog for streaming fixes and E2E tests

Document streaming bug fixes and new E2E test suite in changelog.

**Added:**
- End-to-End test suite documentation (17 tests)
- OpenAI API spec compliance verification
- Tool usage indicators and flexible LLM assertions

**Fixed:**
- Streaming text repetition (delta mode implementation)
- Broken tool execution in streaming
- Invalid schema parameters
- Case sensitivity in model routing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
2025-12-07 15:41:34 +01:00
co-authored by Claude Sonnet 4.5
parent 636d8dcd77
commit 505d284977
+78
View File
@@ -8,6 +8,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
#### Phase 2: The Steward (Two-Tier Architecture)
- **The Steward Agent**: First-tier LLM agent for request analysis and capability recommendation
- Analyzes requests with full conversation context awareness
- Recommends relevant household capabilities for each request
- Detects missing capabilities and provides guidance
- Estimates request complexity (simple/moderate/complex)
- Uses same Ollama model as Tatlock for VRAM efficiency
- **Household Registry**: Centralized capability management system
- `HouseholdRegistry` for registering capabilities and toolsets
- `HouseholdCapability` executive summaries for coordination
- `HouseholdMember` specifications with PydanticAI toolsets
- Domain-based tool organization (e.g., `src/agents/tatlock_core/`)
- Dynamic tool scoping per request
- **Request Preprocessing Pipeline**: Steward → Tatlock flow integration
- `preprocess_request()` orchestrates Steward analysis
- Creates scoped toolsets based on recommendations
- Formats Steward notes for Butler (conversation context included)
- Integrated with Responses API via `create_response_with_steward()`
- **Tool Usage Tracking**: Benchmarking and accuracy analysis
- `ToolCallTracker` for monitoring recommended vs. actual tool usage
- Tracks recommendation accuracy metrics
- Records benchmarks to Redis for cross-session analysis
- Supports precision/recall/F1 score calculation
- **Streaming Transparency**: Real-time Steward analysis visibility
- Streams Steward's reasoning as reasoning summary deltas
- Streams Tatlock's response as output text deltas
- Full SSE support for Steward + Tatlock flow
- Conversation context and missing capabilities visible in stream
- **Structured Logging**: Operation timing and metadata tracking
- `structlog`-based JSON logging for machine parsing
- Context managers for automatic operation timing
- Metadata enrichment for debugging and analysis
- Integrated with benchmark recording
- **Redis Benchmark Storage**: Performance metrics persistence
- Cross-session benchmark storage with 30-day expiry
- Time-series metrics for Steward analysis and tool calls
- Queryable by operation, time range, and metadata
- Support for recommendation accuracy tracking
- **Benchmark Analysis Tools**: Performance analysis CLI
- `scripts/benchmark_analysis.py` for metric analysis
- Steward performance statistics (latency, success rate, recommendations)
- Tool recommendation accuracy analysis (precision, recall, F1)
- Per-tool accuracy breakdown and duration statistics
- **End-to-End Test Suite**: Comprehensive API integration tests
- 17 E2E tests making real HTTP requests to running server
- Tests for Chat Completions, Responses API, and streaming endpoints
- OpenAI API spec compliance verification (format validation)
- Steward preprocessing integration verification
- Error handling tests (404, 422 status codes)
- Flexible assertions for LLM output variance
- Tool usage indicators: 🧮 (calculator), 🔍 (search), 🕐 (datetime)
- Full documentation in `tests/e2e/README.md`
#### Phase 1 Enhancements
- **Conversation history support**: Tatlock now remembers previous turns in multi-turn conversations
- OpenAI-format messages converted to PydanticAI `ModelRequest`/`ModelResponse` objects
- Full conversation context passed to agent via `message_history` parameter
@@ -20,13 +83,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Tool usage visible in `<think>` tags in Open WebUI
### Changed
- **Architecture**: Two-tier request flow (Steward analysis → Tatlock execution)
- **Tool Organization**: Tatlock core tools reorganized into domain directory
- **Tool Scoping**: Tatlock runs with dynamically scoped toolsets per request
- **Responses API**: Integrated Steward preprocessing for all Tatlock requests
- **Streaming**: Enhanced to include Steward reasoning transparency
- Enhanced Tatlock agent with conversation memory capabilities
- All tools now log their usage via `RunContext` dependencies
- Improved debug logging for message history construction
### Fixed
- **Streaming text repetition**: Fixed text accumulation bug causing repetitive output in Open WebUI
- Changed from accumulated text to delta mode (`stream_text(delta=True)`)
- Implemented proper `run_with_scoped_tools_stream()` using PydanticAI's `run_stream()`
- Replaced artificial word-by-word chunking with real LLM deltas
- **Broken tool execution in streaming**: Tools now execute properly in streaming mode
- Previously showed raw JSON function calls instead of executed results
- Now properly streams tool execution results
- **Invalid schema parameter**: Removed invalid `thinking` parameter from `ReasoningOutputItem`
- **Case sensitivity in model routing**: Model comparison now case-insensitive (`.lower()`)
- Conversation context now properly maintained across multiple turns
- Tool usage transparency - users can see exactly what queries/calculations are being performed
- Schema object handling in usage calculation (_calculate_usage reordered isinstance checks)
## [0.2.0] - 2025-12-06