From 505d2849778623781ac1777bfdf058b1b06a4dd7 Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Sun, 7 Dec 2025 15:41:34 +0100 Subject: [PATCH] docs: update changelog for streaming fixes and E2E tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document streaming bug fixes and new E2E test suite in changelog. **Added:** - End-to-End test suite documentation (17 tests) - OpenAI API spec compliance verification - Tool usage indicators and flexible LLM assertions **Fixed:** - Streaming text repetition (delta mode implementation) - Broken tool execution in streaming - Invalid schema parameters - Case sensitivity in model routing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 --- CHANGELOG.md | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e4739b8..bd7bc2c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added + +#### Phase 2: The Steward (Two-Tier Architecture) +- **The Steward Agent**: First-tier LLM agent for request analysis and capability recommendation + - Analyzes requests with full conversation context awareness + - Recommends relevant household capabilities for each request + - Detects missing capabilities and provides guidance + - Estimates request complexity (simple/moderate/complex) + - Uses same Ollama model as Tatlock for VRAM efficiency + +- **Household Registry**: Centralized capability management system + - `HouseholdRegistry` for registering capabilities and toolsets + - `HouseholdCapability` executive summaries for coordination + - `HouseholdMember` specifications with PydanticAI toolsets + - Domain-based tool organization (e.g., `src/agents/tatlock_core/`) + - Dynamic tool scoping per request + +- **Request Preprocessing Pipeline**: Steward → Tatlock flow integration + - `preprocess_request()` orchestrates Steward analysis + - Creates scoped toolsets based on recommendations + - Formats Steward notes for Butler (conversation context included) + - Integrated with Responses API via `create_response_with_steward()` + +- **Tool Usage Tracking**: Benchmarking and accuracy analysis + - `ToolCallTracker` for monitoring recommended vs. actual tool usage + - Tracks recommendation accuracy metrics + - Records benchmarks to Redis for cross-session analysis + - Supports precision/recall/F1 score calculation + +- **Streaming Transparency**: Real-time Steward analysis visibility + - Streams Steward's reasoning as reasoning summary deltas + - Streams Tatlock's response as output text deltas + - Full SSE support for Steward + Tatlock flow + - Conversation context and missing capabilities visible in stream + +- **Structured Logging**: Operation timing and metadata tracking + - `structlog`-based JSON logging for machine parsing + - Context managers for automatic operation timing + - Metadata enrichment for debugging and analysis + - Integrated with benchmark recording + +- **Redis Benchmark Storage**: Performance metrics persistence + - Cross-session benchmark storage with 30-day expiry + - Time-series metrics for Steward analysis and tool calls + - Queryable by operation, time range, and metadata + - Support for recommendation accuracy tracking + +- **Benchmark Analysis Tools**: Performance analysis CLI + - `scripts/benchmark_analysis.py` for metric analysis + - Steward performance statistics (latency, success rate, recommendations) + - Tool recommendation accuracy analysis (precision, recall, F1) + - Per-tool accuracy breakdown and duration statistics + +- **End-to-End Test Suite**: Comprehensive API integration tests + - 17 E2E tests making real HTTP requests to running server + - Tests for Chat Completions, Responses API, and streaming endpoints + - OpenAI API spec compliance verification (format validation) + - Steward preprocessing integration verification + - Error handling tests (404, 422 status codes) + - Flexible assertions for LLM output variance + - Tool usage indicators: 🧮 (calculator), 🔍 (search), 🕐 (datetime) + - Full documentation in `tests/e2e/README.md` + +#### Phase 1 Enhancements - **Conversation history support**: Tatlock now remembers previous turns in multi-turn conversations - OpenAI-format messages converted to PydanticAI `ModelRequest`/`ModelResponse` objects - Full conversation context passed to agent via `message_history` parameter @@ -20,13 +83,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Tool usage visible in `` tags in Open WebUI ### Changed +- **Architecture**: Two-tier request flow (Steward analysis → Tatlock execution) +- **Tool Organization**: Tatlock core tools reorganized into domain directory +- **Tool Scoping**: Tatlock runs with dynamically scoped toolsets per request +- **Responses API**: Integrated Steward preprocessing for all Tatlock requests +- **Streaming**: Enhanced to include Steward reasoning transparency - Enhanced Tatlock agent with conversation memory capabilities - All tools now log their usage via `RunContext` dependencies - Improved debug logging for message history construction ### Fixed +- **Streaming text repetition**: Fixed text accumulation bug causing repetitive output in Open WebUI + - Changed from accumulated text to delta mode (`stream_text(delta=True)`) + - Implemented proper `run_with_scoped_tools_stream()` using PydanticAI's `run_stream()` + - Replaced artificial word-by-word chunking with real LLM deltas +- **Broken tool execution in streaming**: Tools now execute properly in streaming mode + - Previously showed raw JSON function calls instead of executed results + - Now properly streams tool execution results +- **Invalid schema parameter**: Removed invalid `thinking` parameter from `ReasoningOutputItem` +- **Case sensitivity in model routing**: Model comparison now case-insensitive (`.lower()`) - Conversation context now properly maintained across multiple turns - Tool usage transparency - users can see exactly what queries/calculations are being performed +- Schema object handling in usage calculation (_calculate_usage reordered isinstance checks) ## [0.2.0] - 2025-12-06