feat: implement Phase 2 two-tier architecture with Steward

Add comprehensive two-tier architecture where Steward analyzes requests
and Tatlock executes with scoped tools. Includes full infrastructure for
request preprocessing, tool tracking, benchmarking, and streaming.

**Added:**
- Steward agent for request analysis and capability recommendation
- Household Registry for centralized capability management
- Request preprocessing pipeline (Steward → Tatlock flow)
- Tool usage tracking and benchmarking system
- Streaming transparency (Steward reasoning visible in streams)
- Structured logging with operation timing
- Redis benchmark storage with 30-day expiry
- Benchmark analysis CLI tools

**Infrastructure:**
- src/agents/steward/ - Steward agent implementation
- src/agents/tatlock_core/ - Tatlock capability domain
- src/core/preprocessing.py - Request preprocessing pipeline
- src/core/tool_tracking.py - Tool call tracking
- src/core/benchmarks.py - Benchmark recording system
- src/core/household_registry.py - Capability registry
- src/core/startup.py - Application startup coordination
- src/core/logging_config.py - Structured logging setup

**Integration:**
- Responses API uses Steward for Tatlock requests
- Chat Completions wraps Responses API for OpenAI compatibility
- Streaming coordinator supports Steward + Tatlock flow
- Tool scoping per request based on Steward recommendations

**Testing:**
- Integration tests for Steward-Tatlock flow
- Benchmark and registry unit tests
- Steward streaming tests

See PHASE2_PLAN.md and PHASE2_COMPLETE.md for detailed documentation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
2025-12-07 15:39:20 +01:00
co-authored by Claude Sonnet 4.5
parent 2577730546
commit 6eed5f4d13
34 changed files with 6362 additions and 133 deletions
+105
View File
@@ -0,0 +1,105 @@
"""
Request preprocessing pipeline.
Analyzes requests via the Steward and creates scoped toolsets for Tatlock.
"""
from dataclasses import dataclass
from typing import Any, Optional
from src.agents.steward import analyze_request, format_steward_note
from src.agents.steward.schemas import StewardRecommendation
from src.core.household_registry import get_household_registry
from src.core.logging_config import get_logger
logger = get_logger(__name__)
@dataclass
class EnrichedRequest:
"""
Request enriched with Steward's analysis.
Attributes:
original_request: The user's original message
steward_note: Formatted note for Tatlock (includes context analysis)
scoped_tools: List of tools from recommended capabilities
recommendation: Full Steward recommendation
steward_reasoning: Plain text reasoning for streaming to user
"""
original_request: str
steward_note: str
scoped_tools: list[Any] # PydanticAI tool definitions
recommendation: StewardRecommendation
steward_reasoning: str
async def preprocess_request(
user_request: str,
conversation_history: list[dict],
conversation_id: Optional[str] = None,
) -> EnrichedRequest:
"""
Analyze request via Steward and prepare scoped context for Tatlock.
This is the main preprocessing pipeline that:
1. Calls Steward with full conversation history
2. Gets capability recommendations
3. Creates scoped toolset from recommended capabilities
4. Formats a note for Tatlock with context analysis
Args:
user_request: Current user message to analyze
conversation_history: Full conversation history (all previous turns)
conversation_id: Optional conversation ID for tracking
Returns:
EnrichedRequest with scoped tools and Steward analysis
Example:
>>> enriched = await preprocess_request(
... "What's sqrt(144)?",
... conversation_history=[],
... )
>>> print(enriched.recommendation.recommended_capabilities)
['tatlock_core']
>>> print(len(enriched.scoped_tools))
5 # All tatlock_core tools
"""
logger.info(
"preprocessing_request",
request_preview=user_request[:100],
history_length=len(conversation_history),
conversation_id=conversation_id,
)
# Call Steward with full conversation history
recommendation = await analyze_request(
user_request,
conversation_history=conversation_history,
conversation_id=conversation_id,
)
# Format note for Tatlock (includes conversation context)
steward_note = await format_steward_note(recommendation)
# Get scoped tools from household registry
registry = get_household_registry()
scoped_tools = registry.get_scoped_tools(
recommendation.recommended_capabilities
)
logger.info(
"preprocessing_complete",
recommended_capabilities=recommendation.recommended_capabilities,
tool_count=len(scoped_tools),
complexity=recommendation.estimated_complexity,
has_context=recommendation.conversation_context.has_previous_context,
)
return EnrichedRequest(
original_request=user_request,
steward_note=steward_note,
scoped_tools=scoped_tools,
recommendation=recommendation,
steward_reasoning=recommendation.reasoning,
)