docs: update changelog for streaming fixes and E2E tests
Document streaming bug fixes and new E2E test suite in changelog. **Added:** - End-to-End test suite documentation (17 tests) - OpenAI API spec compliance verification - Tool usage indicators and flexible LLM assertions **Fixed:** - Streaming text repetition (delta mode implementation) - Broken tool execution in streaming - Invalid schema parameters - Case sensitivity in model routing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -8,6 +8,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
#### Phase 2: The Steward (Two-Tier Architecture)
|
||||
- **The Steward Agent**: First-tier LLM agent for request analysis and capability recommendation
|
||||
- Analyzes requests with full conversation context awareness
|
||||
- Recommends relevant household capabilities for each request
|
||||
- Detects missing capabilities and provides guidance
|
||||
- Estimates request complexity (simple/moderate/complex)
|
||||
- Uses same Ollama model as Tatlock for VRAM efficiency
|
||||
|
||||
- **Household Registry**: Centralized capability management system
|
||||
- `HouseholdRegistry` for registering capabilities and toolsets
|
||||
- `HouseholdCapability` executive summaries for coordination
|
||||
- `HouseholdMember` specifications with PydanticAI toolsets
|
||||
- Domain-based tool organization (e.g., `src/agents/tatlock_core/`)
|
||||
- Dynamic tool scoping per request
|
||||
|
||||
- **Request Preprocessing Pipeline**: Steward → Tatlock flow integration
|
||||
- `preprocess_request()` orchestrates Steward analysis
|
||||
- Creates scoped toolsets based on recommendations
|
||||
- Formats Steward notes for Butler (conversation context included)
|
||||
- Integrated with Responses API via `create_response_with_steward()`
|
||||
|
||||
- **Tool Usage Tracking**: Benchmarking and accuracy analysis
|
||||
- `ToolCallTracker` for monitoring recommended vs. actual tool usage
|
||||
- Tracks recommendation accuracy metrics
|
||||
- Records benchmarks to Redis for cross-session analysis
|
||||
- Supports precision/recall/F1 score calculation
|
||||
|
||||
- **Streaming Transparency**: Real-time Steward analysis visibility
|
||||
- Streams Steward's reasoning as reasoning summary deltas
|
||||
- Streams Tatlock's response as output text deltas
|
||||
- Full SSE support for Steward + Tatlock flow
|
||||
- Conversation context and missing capabilities visible in stream
|
||||
|
||||
- **Structured Logging**: Operation timing and metadata tracking
|
||||
- `structlog`-based JSON logging for machine parsing
|
||||
- Context managers for automatic operation timing
|
||||
- Metadata enrichment for debugging and analysis
|
||||
- Integrated with benchmark recording
|
||||
|
||||
- **Redis Benchmark Storage**: Performance metrics persistence
|
||||
- Cross-session benchmark storage with 30-day expiry
|
||||
- Time-series metrics for Steward analysis and tool calls
|
||||
- Queryable by operation, time range, and metadata
|
||||
- Support for recommendation accuracy tracking
|
||||
|
||||
- **Benchmark Analysis Tools**: Performance analysis CLI
|
||||
- `scripts/benchmark_analysis.py` for metric analysis
|
||||
- Steward performance statistics (latency, success rate, recommendations)
|
||||
- Tool recommendation accuracy analysis (precision, recall, F1)
|
||||
- Per-tool accuracy breakdown and duration statistics
|
||||
|
||||
- **End-to-End Test Suite**: Comprehensive API integration tests
|
||||
- 17 E2E tests making real HTTP requests to running server
|
||||
- Tests for Chat Completions, Responses API, and streaming endpoints
|
||||
- OpenAI API spec compliance verification (format validation)
|
||||
- Steward preprocessing integration verification
|
||||
- Error handling tests (404, 422 status codes)
|
||||
- Flexible assertions for LLM output variance
|
||||
- Tool usage indicators: 🧮 (calculator), 🔍 (search), 🕐 (datetime)
|
||||
- Full documentation in `tests/e2e/README.md`
|
||||
|
||||
#### Phase 1 Enhancements
|
||||
- **Conversation history support**: Tatlock now remembers previous turns in multi-turn conversations
|
||||
- OpenAI-format messages converted to PydanticAI `ModelRequest`/`ModelResponse` objects
|
||||
- Full conversation context passed to agent via `message_history` parameter
|
||||
@@ -20,13 +83,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- Tool usage visible in `<think>` tags in Open WebUI
|
||||
|
||||
### Changed
|
||||
- **Architecture**: Two-tier request flow (Steward analysis → Tatlock execution)
|
||||
- **Tool Organization**: Tatlock core tools reorganized into domain directory
|
||||
- **Tool Scoping**: Tatlock runs with dynamically scoped toolsets per request
|
||||
- **Responses API**: Integrated Steward preprocessing for all Tatlock requests
|
||||
- **Streaming**: Enhanced to include Steward reasoning transparency
|
||||
- Enhanced Tatlock agent with conversation memory capabilities
|
||||
- All tools now log their usage via `RunContext` dependencies
|
||||
- Improved debug logging for message history construction
|
||||
|
||||
### Fixed
|
||||
- **Streaming text repetition**: Fixed text accumulation bug causing repetitive output in Open WebUI
|
||||
- Changed from accumulated text to delta mode (`stream_text(delta=True)`)
|
||||
- Implemented proper `run_with_scoped_tools_stream()` using PydanticAI's `run_stream()`
|
||||
- Replaced artificial word-by-word chunking with real LLM deltas
|
||||
- **Broken tool execution in streaming**: Tools now execute properly in streaming mode
|
||||
- Previously showed raw JSON function calls instead of executed results
|
||||
- Now properly streams tool execution results
|
||||
- **Invalid schema parameter**: Removed invalid `thinking` parameter from `ReasoningOutputItem`
|
||||
- **Case sensitivity in model routing**: Model comparison now case-insensitive (`.lower()`)
|
||||
- Conversation context now properly maintained across multiple turns
|
||||
- Tool usage transparency - users can see exactly what queries/calculations are being performed
|
||||
- Schema object handling in usage calculation (_calculate_usage reordered isinstance checks)
|
||||
|
||||
## [0.2.0] - 2025-12-06
|
||||
|
||||
|
||||
Reference in New Issue
Block a user