Compare commits

...
1 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Opus 4.5 49f0da8068 feat: two-phase execution, think slugs, query enrichment (v1.6.0)
Build and Push / build (release) Successful in 1m14s
Two-Phase Tatlock Execution:
- orchestrate_tool_calls() for Phase 1 coordination
- synthesize_from_results() for Phase 2 butler-toned synthesis
- Guarantees butler personality in all responses

Automatic Think Slugs:
- Deterministic butler-perspective messages during expert delegation
- ActionType enum: RETRIEVE, RESEARCH, CREATE, CONTROL, RECORD
- HOUSEHOLD_THINK_MESSAGES mapping for all experts
- Streaming delegation wrappers with automatic think messages

Steward Query Enrichment:
- Auto-fill user context (location, timezone) when not specified
- _build_enriched_query() with regex word boundary matching
- enriched_query field in StewardRecommendation schema

Documentation:
- ORCHESTRATION_SCENARIOS.md rewritten with Mermaid diagrams
- New Housekeeper and Biographer scenarios
- TESTING_IMPROVEMENTS.md for future LLM testing patterns

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 14:00:32 +01:00
13 changed files with 1919 additions and 263 deletions
+52 -1
View File
@@ -7,6 +7,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
## [1.6.0] - 2025-12-15
### Added
#### Two-Phase Tatlock Execution
- **Phase 1: Orchestration** - Executes tool calls and expert delegations, returns structured results
- **Phase 2: Synthesis** - Synthesizes butler-toned response from gathered results
- `orchestrate_tool_calls()` method in TatlockAgent for coordination phase
- `synthesize_from_results()` method in TatlockAgent for synthesis phase
- Guarantees butler personality in all responses by separating coordination from response generation
#### Automatic Think Slugs
- **Deterministic butler-perspective messages** during expert delegation (no LLM involved)
- `ActionType` enum: RETRIEVE, RESEARCH, CREATE, CONTROL, RECORD
- `HOUSEHOLD_THINK_MESSAGES` mapping with butler-perspective messages for all experts:
- Librarian: "Allow me to consult the archives, sir." / "I'm having the Librarian prepare a new entry."
- Biographer: "Let me consult the household records." / "I've asked the Biographer to take note, sir."
- Housekeeper: "I'm instructing the household staff now, sir." / "Allow me to inquire with the household staff."
- `_detect_action_type()` function for keyword-based action detection
- `get_think_message()` helper for retrieving appropriate messages
- Streaming delegation wrappers: `stream_delegate_to_librarian()`, `stream_delegate_to_biographer()`, `stream_delegate_to_housekeeper()`
- `STREAMING_DELEGATION_WRAPPERS` mapping in delegation.py
- `get_streaming_delegation_tools()` method in HouseholdRegistry
#### Steward Query Enrichment
- **Auto-fill user context** (location, timezone) when not specified in query
- `_build_enriched_query()` function in steward service
- Regex word boundary matching for accurate location detection (avoids false positives)
- `enriched_query` field added to `StewardRecommendation` schema
- Automatic enrichment for weather queries (location), time queries (timezone), temperature preferences
#### Documentation
- **ORCHESTRATION_SCENARIOS.md** completely rewritten with:
- Mermaid flow diagrams for two-phase execution
- 4 new Housekeeper scenarios (light control, device status, parallel delegation)
- Biographer memory recording scenario
- Complete think slug reference tables
- Action type detection tables
- Updated architecture mindmap
- **TESTING_IMPROVEMENTS.md** - LLM testing best practices for future implementation
### Changed
- `create_response_with_steward()` now uses two-phase execution
- `_direct_delegation()` routes through synthesis phase for consistent butler tone
- `_execute_single_delegation()` now supports housekeeper
- Streaming response handler integrated with think slug system
- All 326 unit tests passing
## [1.5.0] - 2025-12-15 ## [1.5.0] - 2025-12-15
### Added ### Added
@@ -608,7 +657,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- CORS middleware - CORS middleware
- Exception handlers (OpenAI-compatible error format) - Exception handlers (OpenAI-compatible error format)
[Unreleased]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.4.0...main [Unreleased]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.6.0...main
[1.6.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.5.0...v1.6.0
[1.5.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.4.0...v1.5.0
[1.4.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.3...v1.4.0 [1.4.0]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.3...v1.4.0
[1.3.3]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.2...v1.3.3 [1.3.3]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.2...v1.3.3
[1.3.2]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.1...v1.3.2 [1.3.2]: https://git.schweitz.net/jpmschweitzer/tatlock/compare/v1.3.1...v1.3.2
+677 -216
View File
File diff suppressed because it is too large Load Diff
+105
View File
@@ -0,0 +1,105 @@
# Testing Improvements for LLM Outputs
## Problem
LLM outputs are non-deterministic. Tests checking for exact string matches fail when the LLM writes "thirty-seven" instead of "37".
## Proposed Solutions
### 1. LLM-as-Judge Pattern
Use a smaller/faster model to evaluate semantic correctness:
```python
async def llm_judge(output: str, criteria: str) -> bool:
"""Use LLM to evaluate if output meets criteria."""
prompt = f"""
Evaluate if this output is correct:
Output: {output}
Criteria: {criteria}
Answer only YES or NO.
"""
result = await judge_model.run(prompt)
return "YES" in result.output.upper()
# Usage in test:
assert await llm_judge(
response,
"The answer correctly states that sqrt(144) + 25 = 37"
)
```
### 2. Fuzzy/Regex Matching
For numeric answers, accept multiple representations:
```python
import re
def contains_number(text: str, number: int) -> bool:
"""Check if text contains number in any form."""
patterns = [
rf'\b{number}\b', # Digit form
number_to_words(number), # Word form
]
return any(re.search(p, text, re.I) for p in patterns)
# Usage:
assert contains_number(response, 37) # Matches "37" or "thirty-seven"
```
### 3. DeepEval Framework
```python
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
def test_calculation():
test_case = LLMTestCase(
input="What is sqrt(144) + 25?",
actual_output=response,
expected_output="37"
)
metric = AnswerRelevancyMetric(threshold=0.7)
assert metric.measure(test_case)
```
### 4. pytest-evals Plugin
Minimal pytest plugin for LLM testing with metrics collection.
```bash
pip install pytest-evals
```
### 5. Multiple Runs with Threshold
Run flaky tests multiple times and require majority pass:
```python
@pytest.mark.flaky(reruns=3, reruns_delay=1)
def test_llm_response():
...
```
Or custom:
```python
@pytest.mark.parametrize("run", range(3))
def test_llm_response(run):
...
# Aggregate results across runs
```
## Resources
- [DeepEval](https://github.com/confident-ai/deepeval) - LLM evaluation framework
- [pytest-evals](https://github.com/AlmogBaku/pytest-evals) - pytest plugin for LLM evals
- [LLM Testing Guide 2025](https://www.confident-ai.com/blog/llm-testing-in-2024-top-methods-and-strategies)
- [Testing LLM Applications - Langfuse](https://langfuse.com/blog/2025-10-21-testing-llm-applications)
## Implementation Priority
1. Add fuzzy number matching helper (quick win)
2. Evaluate DeepEval for complex output testing
3. Consider LLM-as-judge for semantic correctness
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "tatlock" name = "tatlock"
version = "1.5.0" version = "1.6.0"
description = "OpenAI-compatible API with Ollama backend" description = "OpenAI-compatible API with Ollama backend"
requires-python = ">=3.12" requires-python = ">=3.12"
dependencies = [] dependencies = []
+221 -1
View File
@@ -9,13 +9,136 @@ This implements the agent-as-tool pattern recommended by PydanticAI:
agents call other agents via tool wrappers, keeping each agent focused. agents call other agents via tool wrappers, keeping each agent focused.
""" """
from dataclasses import dataclass, field from dataclasses import dataclass, field
from typing import Callable, Optional, Any from enum import Enum
from typing import AsyncGenerator, Callable, Optional, Any
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
# =============================================================================
# Action Types for Think Slug Selection
# =============================================================================
class ActionType(Enum):
"""
Categories of actions for selecting appropriate think messages.
Each expert has different action types that warrant different
butler-perspective messages to the user.
"""
RETRIEVE = "retrieve" # Looking up existing information
RESEARCH = "research" # Conducting new research (web search, etc.)
CREATE = "create" # Creating new content (pages, notes)
CONTROL = "control" # Controlling devices/automations
RECORD = "record" # Recording memories/notes
# =============================================================================
# Household Think Messages (Butler's Perspective)
# =============================================================================
HOUSEHOLD_THINK_MESSAGES: dict[str, dict[ActionType, dict[str, str]]] = {
"librarian": {
ActionType.RETRIEVE: {
"start": "<think>Allow me to consult the archives, sir.</think>",
"success": "<think>The Librarian has compiled the relevant findings.</think>",
"error": "<think>I'm afraid the archives proved difficult to access.</think>",
},
ActionType.RESEARCH: {
"start": "<think>I've dispatched the Librarian to conduct some fresh research.</think>",
"success": "<think>The Librarian has returned with findings, sir.</think>",
"error": "<think>The research proved inconclusive, I'm afraid.</think>",
},
ActionType.CREATE: {
"start": "<think>I'm having the Librarian prepare a new entry.</think>",
"success": "<think>The new material has been properly catalogued, sir.</think>",
"error": "<think>I'm afraid there was difficulty filing the entry.</think>",
},
},
"biographer": {
ActionType.RETRIEVE: {
"start": "<think>Let me consult the household records.</think>",
"success": "<think>The Biographer has located the relevant information, sir.</think>",
"error": "<think>I'm unable to locate those particular records.</think>",
},
ActionType.RECORD: {
"start": "<think>I've asked the Biographer to take note of this, sir.</think>",
"success": "<think>The household records have been updated accordingly.</think>",
"error": "<think>I'm afraid there was difficulty recording the entry.</think>",
},
},
"housekeeper": {
ActionType.RETRIEVE: {
"start": "<think>Allow me to inquire with the household staff.</think>",
"success": "<think>The staff reports the current status, sir.</think>",
"error": "<think>The household staff is momentarily unavailable, I'm afraid.</think>",
},
ActionType.CONTROL: {
"start": "<think>I'm instructing the household staff now, sir.</think>",
"success": "<think>The household has been configured as requested.</think>",
"error": "<think>I'm afraid the staff reports an issue with that request.</think>",
},
},
}
def _detect_action_type(expert: str, task: str) -> ActionType:
"""
Detect action type from expert name and task description.
Used to select appropriate butler-perspective think messages.
Args:
expert: Name of the expert (librarian, biographer, housekeeper)
task: Task description
Returns:
ActionType: Detected action type for message selection
"""
task_lower = task.lower()
if expert == "librarian":
if any(w in task_lower for w in ["search", "find", "look up", "research"]):
if any(w in task_lower for w in ["web", "online", "internet"]):
return ActionType.RESEARCH
return ActionType.RETRIEVE
if any(w in task_lower for w in ["create", "write", "add", "make", "new"]):
return ActionType.CREATE
return ActionType.RETRIEVE
elif expert == "biographer":
if any(w in task_lower for w in ["remember", "note", "record", "save", "store"]):
return ActionType.RECORD
return ActionType.RETRIEVE
elif expert == "housekeeper":
if any(w in task_lower for w in ["turn", "set", "activate", "enable", "disable", "toggle"]):
return ActionType.CONTROL
return ActionType.RETRIEVE
return ActionType.RETRIEVE
def get_think_message(expert: str, task: str, phase: str) -> str:
"""
Get the appropriate think message for an expert delegation.
Args:
expert: Name of the expert
task: Task description (used to detect action type)
phase: One of "start", "success", "error"
Returns:
str: Butler-perspective think message
"""
action_type = _detect_action_type(expert, task)
expert_messages = HOUSEHOLD_THINK_MESSAGES.get(expert, {})
action_messages = expert_messages.get(action_type, expert_messages.get(ActionType.RETRIEVE, {}))
return action_messages.get(phase, f"<think>Consulting {expert}...</think>")
@dataclass @dataclass
class DelegationTask: class DelegationTask:
""" """
@@ -301,6 +424,103 @@ async def delegate_to_housekeeper(
) )
# =============================================================================
# Streaming Delegation Wrappers (with Think Messages)
# =============================================================================
async def stream_delegate_to_librarian(
task: str,
context: str = "",
) -> AsyncGenerator[str, None]:
"""
Stream delegation to Librarian with automatic think messages.
Yields butler-perspective think messages before and after the delegation,
allowing the UI to show progress to the user.
Args:
task: Task description
context: Additional context
Yields:
str: Think messages and final result marker
"""
# Yield start message (deterministic)
yield get_think_message("librarian", task, "start") + "\n"
# Execute delegation
result = await delegate_to_librarian(task, context)
# Yield completion message (deterministic)
if result.success:
yield get_think_message("librarian", task, "success") + "\n"
else:
yield get_think_message("librarian", task, "error") + "\n"
# Yield result marker for extraction
yield f"__DELEGATION_RESULT__:librarian:{result.output}"
async def stream_delegate_to_biographer(
task: str,
context: str = "",
) -> AsyncGenerator[str, None]:
"""
Stream delegation to Biographer with automatic think messages.
Args:
task: Task description
context: Additional context
Yields:
str: Think messages and final result marker
"""
yield get_think_message("biographer", task, "start") + "\n"
result = await delegate_to_biographer(task, context)
if result.success:
yield get_think_message("biographer", task, "success") + "\n"
else:
yield get_think_message("biographer", task, "error") + "\n"
yield f"__DELEGATION_RESULT__:biographer:{result.output}"
async def stream_delegate_to_housekeeper(
task: str,
context: str = "",
) -> AsyncGenerator[str, None]:
"""
Stream delegation to Housekeeper with automatic think messages.
Args:
task: Task description
context: Additional context
Yields:
str: Think messages and final result marker
"""
yield get_think_message("housekeeper", task, "start") + "\n"
result = await delegate_to_housekeeper(task, context)
if result.success:
yield get_think_message("housekeeper", task, "success") + "\n"
else:
yield get_think_message("housekeeper", task, "error") + "\n"
yield f"__DELEGATION_RESULT__:housekeeper:{result.output}"
# Mapping of streaming delegation wrappers
STREAMING_DELEGATION_WRAPPERS = {
"librarian": stream_delegate_to_librarian,
"biographer": stream_delegate_to_biographer,
"housekeeper": stream_delegate_to_housekeeper,
}
# Future expert delegation wrappers will be added here: # Future expert delegation wrappers will be added here:
# - delegate_to_developer(task, context) -> DelegationResult # - delegate_to_developer(task, context) -> DelegationResult
# - delegate_to_secretary(task, context) -> DelegationResult # - delegate_to_secretary(task, context) -> DelegationResult
+4
View File
@@ -60,6 +60,10 @@ class StewardRecommendation(BaseModel):
default_factory=dict, default_factory=dict,
description="Pre-fetched user context from memory (profile, preferences)" description="Pre-fetched user context from memory (profile, preferences)"
) )
enriched_query: str = Field(
default="",
description="User query with auto-filled context (location, timezone) when not specified"
)
def format_for_butler(self) -> str: def format_for_butler(self) -> str:
""" """
+66
View File
@@ -149,6 +149,68 @@ def _extract_missing_capabilities(text: str) -> Optional[str]:
return None return None
def _build_enriched_query(user_request: str, memory_context: dict[str, Any]) -> str:
"""
Build an enriched query by appending user context when not specified.
When the user asks location-dependent questions (weather, nearby, etc.)
without specifying a location, this appends their known location.
Similarly for timezone-dependent queries.
Args:
user_request: The user's original request
memory_context: Pre-fetched memory context with profile/preferences
Returns:
str: Query with context appended, or original query if no enrichment needed
Example:
>>> query = _build_enriched_query(
... "What's the weather?",
... {"profile": {"location": "Amsterdam", "timezone": "Europe/Amsterdam"}}
... )
>>> query
"What's the weather?\n\n[User Context: location=Amsterdam, timezone=Europe/Amsterdam]"
"""
if not memory_context:
return user_request
request_lower = user_request.lower()
profile = memory_context.get("profile", {})
preferences = memory_context.get("preferences", {})
context_parts = []
# Check if location is needed and not specified
location_keywords = ["weather", "temperature", "forecast", "nearby", "local", "here"]
# Use word boundary pattern to avoid false positives like "at" in "what"
location_prepositions = [r'\bin\b', r'\bat\b', r'\bnear\b', r'\baround\b', r'\bfor\b']
location_specified = any(re.search(p, request_lower) for p in location_prepositions)
if any(word in request_lower for word in location_keywords):
if not location_specified and profile.get("location"):
context_parts.append(f"location={profile['location']}")
# Check if timezone is needed and not specified
time_keywords = ["time", "schedule", "meeting", "appointment", "when", "today", "tomorrow"]
timezone_specified = any(word in request_lower for word in ["timezone", "tz", "utc", "gmt"])
if any(word in request_lower for word in time_keywords):
if not timezone_specified and profile.get("timezone"):
context_parts.append(f"timezone={profile['timezone']}")
# Add preferences if relevant
if preferences.get("temperature_unit") and "weather" in request_lower:
context_parts.append(f"temperature_unit={preferences['temperature_unit']}")
# Build enriched query
if context_parts:
context_str = ", ".join(context_parts)
return f"{user_request}\n\n[User Context: {context_str}]"
return user_request
async def _prefetch_memory_context(user_request: str) -> dict[str, Any]: async def _prefetch_memory_context(user_request: str) -> dict[str, Any]:
""" """
Pre-fetch user context that might be needed for this request. Pre-fetch user context that might be needed for this request.
@@ -277,6 +339,9 @@ async def analyze_request(
context = _extract_conversation_context(analysis_text, conversation_history) context = _extract_conversation_context(analysis_text, conversation_history)
missing = _extract_missing_capabilities(analysis_text) missing = _extract_missing_capabilities(analysis_text)
# Build enriched query with auto-filled context
enriched_query = _build_enriched_query(user_request, memory_context)
recommendation = StewardRecommendation( recommendation = StewardRecommendation(
recommended_capabilities=capabilities, recommended_capabilities=capabilities,
reasoning=analysis_text, reasoning=analysis_text,
@@ -284,6 +349,7 @@ async def analyze_request(
conversation_context=context, conversation_context=context,
missing_capabilities=missing, missing_capabilities=missing,
memory_context=memory_context, memory_context=memory_context,
enriched_query=enriched_query,
) )
# Update log context with results # Update log context with results
+235
View File
@@ -630,6 +630,241 @@ class TatlockAgent(AgentInterface):
logger.info("tatlock_scoped_run_complete") logger.info("tatlock_scoped_run_complete")
async def orchestrate_tool_calls(
self,
user_message: str,
steward_note: str,
scoped_tools: list[Any],
message_history: list[dict],
tool_tracker: Any = None,
) -> dict[str, Any]:
"""
Phase 1: Execute tool calls and delegations, return structured results.
This is the coordination phase where Tatlock orchestrates tool calls
and expert delegations. The raw output is captured for Phase 2 synthesis.
Args:
user_message: The user's original message
steward_note: Note from Steward (invisible to user)
scoped_tools: List of tool definitions from household registry
message_history: Conversation history
tool_tracker: Optional tool call tracker for benchmarking
Returns:
dict with:
- tools_called: List of tool names that were called
- expert_results: Dict mapping expert names to their outputs
- tool_outputs: Dict mapping tool names to their outputs
- raw_output: The agent's raw text output
"""
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.ollama import OllamaProvider
from pydantic_ai.settings import ModelSettings
from pydantic_ai.messages import (
ModelRequest,
ModelResponse,
UserPromptPart,
TextPart,
ToolCallPart,
ToolReturnPart,
)
logger.info(
"tatlock_orchestrate_tool_calls",
user_message_preview=user_message[:100],
scoped_tool_count=len(scoped_tools),
history_length=len(message_history),
)
# Create a fresh agent instance with scoped tools only
clean_host = self.ollama_host.rstrip('/')
base_url = f"{clean_host}/v1"
ollama_model = OpenAIChatModel(
model_name=self.model_name,
provider=OllamaProvider(base_url=base_url)
)
# Create agent with scoped tools
scoped_agent = Agent(
ollama_model,
system_prompt=TATLOCK_SYSTEM_PROMPT,
tools=scoped_tools,
)
# Prepend Steward's note to the request
enriched_message = f"{steward_note}\n\n{user_message}"
# Convert message history to PydanticAI format
pydantic_history = []
for msg in message_history:
role = msg.get("role")
content = msg.get("content", "")
if not content or not content.strip():
continue
if role == "user":
pydantic_history.append(
ModelRequest(parts=[UserPromptPart(content=content)])
)
elif role == "assistant":
pydantic_history.append(
ModelResponse(parts=[TextPart(content=content)])
)
# Run with scoped tools and tracker
result = await scoped_agent.run(
enriched_message,
message_history=pydantic_history if pydantic_history else None,
deps=tool_tracker,
model_settings=ModelSettings(extra_body={"tool_choice": "required"})
)
# Extract tool calls and results from the agent's messages
tools_called = []
expert_results = {}
tool_outputs = {}
# Parse through new messages to find tool calls and returns
for msg in result.new_messages():
if isinstance(msg, ModelResponse):
for part in msg.parts:
if isinstance(part, ToolCallPart):
tools_called.append(part.tool_name)
elif isinstance(msg, ModelRequest):
for part in msg.parts:
if isinstance(part, ToolReturnPart):
tool_name = part.tool_name
content = part.content
# Categorize as expert result or tool output
if tool_name.startswith("delegate_to_"):
expert_name = tool_name.replace("delegate_to_", "")
expert_results[expert_name] = content
else:
tool_outputs[tool_name] = content
logger.info(
"tatlock_orchestration_complete",
tools_called=tools_called,
expert_count=len(expert_results),
tool_output_count=len(tool_outputs),
)
return {
"tools_called": tools_called,
"expert_results": expert_results,
"tool_outputs": tool_outputs,
"raw_output": result.output,
}
async def synthesize_from_results(
self,
user_message: str,
orchestration_results: dict[str, Any],
message_history: list[dict],
) -> str:
"""
Phase 2: Synthesize butler-toned response from gathered results.
This is the synthesis phase where Tatlock takes the coordination
results and produces a properly butler-toned response.
Args:
user_message: The user's original message
orchestration_results: Results from orchestrate_tool_calls()
message_history: Conversation history
Returns:
str: Butler-toned response synthesized from all results
"""
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.ollama import OllamaProvider
from pydantic_ai.messages import ModelRequest, ModelResponse, UserPromptPart, TextPart
logger.info(
"tatlock_synthesize_from_results",
user_message_preview=user_message[:100],
expert_count=len(orchestration_results.get("expert_results", {})),
tool_count=len(orchestration_results.get("tool_outputs", {})),
)
# Build synthesis prompt with all available information
synthesis_parts = []
synthesis_parts.append(f"The user asked: {user_message}")
synthesis_parts.append("")
# Add expert findings if any
if orchestration_results.get("expert_results"):
synthesis_parts.append("Expert findings:")
for expert, result in orchestration_results["expert_results"].items():
synthesis_parts.append(f"- {expert.title()}: {result}")
synthesis_parts.append("")
# Add tool outputs if any
if orchestration_results.get("tool_outputs"):
synthesis_parts.append("Tool results:")
for tool, result in orchestration_results["tool_outputs"].items():
synthesis_parts.append(f"- {tool}: {result}")
synthesis_parts.append("")
synthesis_parts.append(
"Based on this information, provide a response to the user. "
"Maintain your butler personality - address them as 'sir', "
"use formal but personable language, and be helpful."
)
synthesis_prompt = "\n".join(synthesis_parts)
# Create synthesis agent (no tools needed)
clean_host = self.ollama_host.rstrip('/')
base_url = f"{clean_host}/v1"
ollama_model = OpenAIChatModel(
model_name=self.model_name,
provider=OllamaProvider(base_url=base_url)
)
# Synthesis agent uses butler prompt but no tools
synthesis_agent = Agent(
ollama_model,
system_prompt=TATLOCK_SYSTEM_PROMPT,
# No tools for synthesis phase
)
# Convert message history to PydanticAI format
pydantic_history = []
for msg in message_history:
role = msg.get("role")
content = msg.get("content", "")
if not content or not content.strip():
continue
if role == "user":
pydantic_history.append(
ModelRequest(parts=[UserPromptPart(content=content)])
)
elif role == "assistant":
pydantic_history.append(
ModelResponse(parts=[TextPart(content=content)])
)
# Run synthesis
result = await synthesis_agent.run(
synthesis_prompt,
message_history=pydantic_history if pydantic_history else None,
)
logger.info(
"tatlock_synthesis_complete",
response_preview=result.output[:100],
)
return result.output
async def get_capabilities(self) -> dict: async def get_capabilities(self) -> dict:
"""Return current capabilities.""" """Return current capabilities."""
return { return {
+59
View File
@@ -273,6 +273,65 @@ class HouseholdRegistry:
return tools return tools
def get_streaming_delegation_tools(self, names: list[str]) -> list[Any]:
"""
Get streaming delegation wrapper tools for specified capabilities.
Similar to get_delegation_tools() but returns streaming wrappers
that yield butler-perspective think messages during execution.
These wrappers emit think slugs like:
- "Allow me to consult the archives, sir."
- "The Librarian has compiled the relevant findings."
Args:
names: List of member names to include
Returns:
List of streaming delegation wrappers and/or raw tools
Example:
>>> tools = registry.get_streaming_delegation_tools(["librarian"])
>>> async for chunk in tools[0](task="Search for Docker"):
... print(chunk) # Yields think messages then result
"""
from src.agents.delegation import STREAMING_DELEGATION_WRAPPERS
tools = []
for name in names:
member = self._members.get(name)
if not member:
logger.warning(
"household_member_not_found",
requested_name=name,
available_names=list(self._members.keys()),
)
continue
# Check if this member has a streaming delegation wrapper
if name in STREAMING_DELEGATION_WRAPPERS and member.agent is not None:
tools.append(STREAMING_DELEGATION_WRAPPERS[name])
logger.debug(
"streaming_delegation_wrapper_added",
member=name,
)
else:
# No agent = direct tools (e.g., tatlock_core)
tools.extend(member.tools)
logger.debug(
"raw_tools_added",
member=name,
tool_count=len(member.tools),
)
logger.info(
"streaming_delegation_tools_created",
requested_members=names,
total_tools=len(tools),
)
return tools
def list_members(self) -> list[str]: def list_members(self) -> list[str]:
""" """
List all registered member names. List all registered member names.
+105 -24
View File
@@ -43,7 +43,7 @@ async def _execute_single_delegation(
Execute a single delegation to an agent. Execute a single delegation to an agent.
Args: Args:
agent_name: Name of agent (biographer, librarian) agent_name: Name of agent (biographer, librarian, housekeeper)
task: Task description task: Task description
tracker: Tool call tracker tracker: Tool call tracker
@@ -67,6 +67,13 @@ async def _execute_single_delegation(
await tracker.track_call("delegate_to_librarian", duration) await tracker.track_call("delegate_to_librarian", duration)
return (agent_name, result.output) return (agent_name, result.output)
elif agent_name == "housekeeper":
from src.agents.delegation import delegate_to_housekeeper
result = await delegate_to_housekeeper(task=task)
duration = time.time() - start_time
await tracker.track_call("delegate_to_housekeeper", duration)
return (agent_name, result.output)
else: else:
return (agent_name, f"Unknown agent: {agent_name}") return (agent_name, f"Unknown agent: {agent_name}")
@@ -244,6 +251,68 @@ async def _direct_delegation(
return "\n\n".join(results) if results else "I apologize, sir. No delegation results available." return "\n\n".join(results) if results else "I apologize, sir. No delegation results available."
async def _direct_delegation_with_results(
user_message: str,
recommendation: "StewardRecommendation",
tracker: "ToolCallTracker",
conversation_id: str,
) -> dict:
"""
Directly delegate to expert agents and return structured results.
This is the Phase 1 variant of direct delegation that returns results
in the same format as TatlockAgent.orchestrate_tool_calls() for
consistent Phase 2 synthesis.
Args:
user_message: User's request
recommendation: Steward's recommendation
tracker: Tool call tracker
conversation_id: Conversation ID
Returns:
dict: Orchestration results with expert_results, tool_outputs, etc.
"""
logger.info(
"direct_delegation_with_results",
agents=recommendation.recommended_capabilities,
conversation_id=conversation_id,
)
expert_results = {}
tools_called = []
for agent in recommendation.recommended_capabilities:
try:
agent_name, result = await _execute_single_delegation(
agent, user_message, tracker
)
expert_results[agent_name] = result
tools_called.append(f"delegate_to_{agent_name}")
logger.info(
"direct_delegation_result",
agent=agent_name,
result_preview=result[:100] if result else "empty",
conversation_id=conversation_id,
)
except Exception as e:
logger.error(
"direct_delegation_failed",
agent=agent,
error=str(e),
conversation_id=conversation_id,
)
expert_results[agent] = f"Error: {e}"
return {
"tools_called": tools_called,
"expert_results": expert_results,
"tool_outputs": {}, # No tool outputs for direct delegation
"raw_output": "", # No raw output for direct delegation
}
# Global conversation history tracker # Global conversation history tracker
# In production, this would be backed by a database or Redis # In production, this would be backed by a database or Redis
_conversation_history = ConversationHistory(max_turns=20) _conversation_history = ConversationHistory(max_turns=20)
@@ -379,12 +448,13 @@ async def create_response(request: ResponseRequest) -> Response:
async def create_response_with_steward(request: ResponseRequest) -> Response: async def create_response_with_steward(request: ResponseRequest) -> Response:
""" """
Create response using Steward preprocessing (Phase 2 flow). Create response using Steward preprocessing and two-phase Tatlock execution.
This is the two-tier architecture where: This is the two-tier architecture with two-phase synthesis:
1. Steward analyzes the request and recommends capabilities 1. Steward analyzes the request and recommends capabilities
2. Tatlock runs with scoped tools based on recommendations 2. Phase 1: Tatlock orchestrates tool calls and expert delegations
3. Tool usage is tracked for benchmarking 3. Phase 2: Tatlock synthesizes butler-toned response from results
4. Tool usage is tracked for benchmarking
Args: Args:
request: Response request request: Response request
@@ -420,37 +490,39 @@ async def create_response_with_steward(request: ResponseRequest) -> Response:
conversation_id=conversation_id, conversation_id=conversation_id,
) )
# Phase 1: Steward preprocessing # Steward preprocessing
enriched = await preprocess_request( enriched = await preprocess_request(
user_message, user_message,
conversation_history=conversation_history, conversation_history=conversation_history,
conversation_id=conversation_id, conversation_id=conversation_id,
) )
# Phase 2: Initialize tool tracker # Initialize tool tracker
tracker = ToolCallTracker( tracker = ToolCallTracker(
recommended_capabilities=enriched.recommendation.recommended_capabilities, recommended_capabilities=enriched.recommendation.recommended_capabilities,
conversation_id=conversation_id, conversation_id=conversation_id,
) )
# Phase 3: Check if direct delegation is recommended # Check if direct delegation is recommended
# If Steward recommends ONLY delegation agents (biographer/librarian), # If Steward recommends ONLY delegation agents (biographer/librarian/housekeeper),
# skip Tatlock and delegate directly # we still use two-phase but delegate directly in Phase 1
delegation_agents = {"biographer", "librarian", "housekeeper"}
delegation_only = all( delegation_only = all(
cap in ("biographer", "librarian") cap in delegation_agents
for cap in enriched.recommendation.recommended_capabilities for cap in enriched.recommendation.recommended_capabilities
) and enriched.recommendation.recommended_capabilities ) and enriched.recommendation.recommended_capabilities
from src.agents.tatlock import TatlockAgent
tatlock = TatlockAgent()
if delegation_only: if delegation_only:
tatlock_response = await _direct_delegation( # Direct delegation path - collect results then synthesize
orchestration_results = await _direct_delegation_with_results(
user_message, enriched.recommendation, tracker, conversation_id user_message, enriched.recommendation, tracker, conversation_id
) )
else: else:
# Phase 3a: Run Tatlock with scoped tools # Phase 1: Orchestrate tool calls
from src.agents.tatlock import TatlockAgent orchestration_results = await tatlock.orchestrate_tool_calls(
tatlock = TatlockAgent()
tatlock_response = await tatlock.run_with_scoped_tools(
user_message=user_message, user_message=user_message,
steward_note=enriched.steward_note, steward_note=enriched.steward_note,
scoped_tools=enriched.scoped_tools, scoped_tools=enriched.scoped_tools,
@@ -458,14 +530,23 @@ async def create_response_with_steward(request: ResponseRequest) -> Response:
tool_tracker=tracker, tool_tracker=tracker,
) )
# Phase 3b: Check for text-based delegation fallback # Handle text-based delegation fallback if present
# If Tatlock outputs [DELEGATE:...] instead of calling the function, if "[DELEGATE:" in orchestration_results.get("raw_output", ""):
# we parse and execute it here text_delegation_results = await _handle_text_delegation(
tatlock_response = await _handle_text_delegation( orchestration_results["raw_output"], tracker, conversation_id
tatlock_response, tracker, conversation_id )
) # Add text delegation results to expert_results
if text_delegation_results != orchestration_results["raw_output"]:
orchestration_results["expert_results"]["text_delegation"] = text_delegation_results
# Phase 4: Finalize tool tracking # Phase 2: Synthesize butler-toned response from all results
tatlock_response = await tatlock.synthesize_from_results(
user_message=user_message,
orchestration_results=orchestration_results,
message_history=conversation_history,
)
# Finalize tool tracking
await tracker.finalize() await tracker.finalize()
# Build response output items # Build response output items
+134 -19
View File
@@ -118,11 +118,12 @@ class StreamingCoordinator:
request: "ResponseRequest" # type: ignore # Forward reference request: "ResponseRequest" # type: ignore # Forward reference
) -> AsyncGenerator[StreamEvent, None]: ) -> AsyncGenerator[StreamEvent, None]:
""" """
Stream response with Steward preprocessing (Phase 2 flow). Stream response with Steward preprocessing and two-phase Tatlock execution.
Streams in order: Streams in order:
1. Steward's analysis as reasoning summary 1. Steward's analysis as reasoning summary
2. Tatlock's response as output text 2. Think slugs during expert delegation (butler-perspective messages)
3. Synthesized butler-toned response as output text
Args: Args:
request: Response request request: Response request
@@ -130,11 +131,17 @@ class StreamingCoordinator:
Yields: Yields:
StreamEvent: Stream of SSE events StreamEvent: Stream of SSE events
""" """
from src.responses.service import _calculate_usage, generate_id, _conversation_history from src.responses.service import (
_calculate_usage,
generate_id,
_conversation_history,
_direct_delegation_with_results,
)
from src.core.preprocessing import preprocess_request from src.core.preprocessing import preprocess_request
from src.core.tool_tracking import ToolCallTracker from src.core.tool_tracking import ToolCallTracker
from src.responses.schemas import MessageOutputItem, ReasoningOutputItem, OutputTextContent from src.responses.schemas import MessageOutputItem, ReasoningOutputItem, OutputTextContent
from src.agents.tatlock import TatlockAgent from src.agents.tatlock import TatlockAgent
from src.agents.delegation import get_think_message, STREAMING_DELEGATION_WRAPPERS
import asyncio import asyncio
output_items = [] output_items = []
@@ -152,7 +159,7 @@ class StreamingCoordinator:
conversation_history = request.input[:-1] if len(request.input) > 1 else [] conversation_history = request.input[:-1] if len(request.input) > 1 else []
# Phase 1: Steward preprocessing # Steward preprocessing
enriched = await preprocess_request( enriched = await preprocess_request(
user_message, user_message,
conversation_history=conversation_history, conversation_history=conversation_history,
@@ -179,31 +186,60 @@ class StreamingCoordinator:
) )
output_items.append(reasoning_item) output_items.append(reasoning_item)
# Phase 2: Initialize tool tracker # Initialize tool tracker
tracker = ToolCallTracker( tracker = ToolCallTracker(
recommended_capabilities=enriched.recommendation.recommended_capabilities, recommended_capabilities=enriched.recommendation.recommended_capabilities,
conversation_id=conversation_id, conversation_id=conversation_id,
) )
# Phase 3: Stream Tatlock's response with scoped tools # Check if direct delegation is recommended
tatlock = TatlockAgent() delegation_agents = {"biographer", "librarian", "housekeeper"}
tatlock_response_parts = [] delegation_only = all(
cap in delegation_agents
for cap in enriched.recommendation.recommended_capabilities
) and enriched.recommendation.recommended_capabilities
async for chunk in tatlock.run_with_scoped_tools_stream( tatlock = TatlockAgent()
if delegation_only:
# Direct delegation path with streaming think slugs
orchestration_results = await self._stream_direct_delegation(
user_message=user_message,
recommendation=enriched.recommendation,
tracker=tracker,
conversation_id=conversation_id,
)
# Stream think slugs that were collected during delegation
for think_msg in orchestration_results.get("think_messages", []):
yield ReasoningSummaryDelta(delta=think_msg)
await asyncio.sleep(0.05)
else:
# Phase 1: Orchestrate tool calls
orchestration_results = await tatlock.orchestrate_tool_calls(
user_message=user_message,
steward_note=enriched.steward_note,
scoped_tools=enriched.scoped_tools,
message_history=conversation_history,
tool_tracker=tracker,
)
# Phase 2: Synthesize butler-toned response
tatlock_response = await tatlock.synthesize_from_results(
user_message=user_message, user_message=user_message,
steward_note=enriched.steward_note, orchestration_results=orchestration_results,
scoped_tools=enriched.scoped_tools,
message_history=conversation_history, message_history=conversation_history,
tool_tracker=tracker, )
):
tatlock_response_parts.append(chunk) # Stream the synthesized response
yield OutputTextDelta(delta=chunk) chunk_size = 50
for i in range(0, len(tatlock_response), chunk_size):
yield OutputTextDelta(delta=tatlock_response[i:i + chunk_size])
await asyncio.sleep(0.02)
yield OutputTextDone() yield OutputTextDone()
# Combine response for output item
tatlock_response = "".join(tatlock_response_parts)
# Add Tatlock message to output items # Add Tatlock message to output items
message_item = MessageOutputItem( message_item = MessageOutputItem(
id=f"msg_{generate_id()}", id=f"msg_{generate_id()}",
@@ -217,7 +253,7 @@ class StreamingCoordinator:
) )
output_items.append(message_item) output_items.append(message_item)
# Phase 4: Finalize tool tracking # Finalize tool tracking
await tracker.finalize() await tracker.finalize()
# Calculate usage and build final response # Calculate usage and build final response
@@ -241,6 +277,85 @@ class StreamingCoordinator:
# Stream error event # Stream error event
yield self._create_error_event(e) yield self._create_error_event(e)
async def _stream_direct_delegation(
self,
user_message: str,
recommendation: "StewardRecommendation", # type: ignore
tracker: "ToolCallTracker", # type: ignore
conversation_id: str,
) -> dict:
"""
Execute direct delegation with streaming think messages.
Collects think messages as delegations execute for streaming to client.
Args:
user_message: User's request
recommendation: Steward's recommendation
tracker: Tool call tracker
conversation_id: Conversation ID
Returns:
dict: Orchestration results with think_messages list
"""
from src.agents.delegation import (
get_think_message,
delegate_to_librarian,
delegate_to_biographer,
delegate_to_housekeeper,
)
import time as time_module
expert_results = {}
tools_called = []
think_messages = []
for agent in recommendation.recommended_capabilities:
# Emit start think message
start_msg = get_think_message(agent, user_message, "start")
think_messages.append(start_msg + "\n")
start_time = time_module.time()
try:
# Execute delegation
if agent == "librarian":
result = await delegate_to_librarian(task=user_message)
elif agent == "biographer":
result = await delegate_to_biographer(task=user_message)
elif agent == "housekeeper":
result = await delegate_to_housekeeper(task=user_message)
else:
result = None
duration = time_module.time() - start_time
await tracker.track_call(f"delegate_to_{agent}", duration)
if result and result.success:
expert_results[agent] = result.output
tools_called.append(f"delegate_to_{agent}")
# Emit success think message
success_msg = get_think_message(agent, user_message, "success")
think_messages.append(success_msg + "\n")
else:
error_msg = result.error if result else "Unknown error"
expert_results[agent] = f"Error: {error_msg}"
# Emit error think message
error_think = get_think_message(agent, user_message, "error")
think_messages.append(error_think + "\n")
except Exception as e:
expert_results[agent] = f"Error: {e}"
error_think = get_think_message(agent, user_message, "error")
think_messages.append(error_think + "\n")
return {
"tools_called": tools_called,
"expert_results": expert_results,
"tool_outputs": {},
"raw_output": "",
"think_messages": think_messages,
}
async def stream_response( async def stream_response(
self, self,
request: "ResponseRequest" # type: ignore # Forward reference request: "ResponseRequest" # type: ignore # Forward reference
+100 -1
View File
@@ -8,7 +8,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
import pytest import pytest
from src.agents.steward.schemas import ConversationContext, StewardRecommendation from src.agents.steward.schemas import ConversationContext, StewardRecommendation
from src.agents.steward.service import analyze_request, format_steward_note from src.agents.steward.service import analyze_request, format_steward_note, _build_enriched_query
from src.core.startup import initialize_application from src.core.startup import initialize_application
@@ -199,3 +199,102 @@ class TestFormatStewardNote:
assert "⚠️ Missing:" in note assert "⚠️ Missing:" in note
assert "Advanced research" in note assert "Advanced research" in note
@pytest.mark.unit
class TestBuildEnrichedQuery:
"""Tests for _build_enriched_query function."""
def test_no_enrichment_without_context(self):
"""Test no enrichment when memory context is empty."""
query = "What's the weather?"
result = _build_enriched_query(query, {})
assert result == query
def test_enrichment_adds_location(self):
"""Test location is appended for weather queries."""
query = "What's the weather?"
memory_context = {
"profile": {"location": "Amsterdam", "timezone": "Europe/Amsterdam"}
}
result = _build_enriched_query(query, memory_context)
assert "location=Amsterdam" in result
assert query in result
assert "[User Context:" in result
def test_no_location_when_specified(self):
"""Test location is not appended when already specified."""
query = "What's the weather in London?"
memory_context = {
"profile": {"location": "Amsterdam"}
}
result = _build_enriched_query(query, memory_context)
# Should not add Amsterdam since location is specified
assert result == query
def test_enrichment_adds_timezone(self):
"""Test timezone is appended for time queries."""
query = "What time is it?"
memory_context = {
"profile": {"timezone": "Europe/Amsterdam"}
}
result = _build_enriched_query(query, memory_context)
assert "timezone=Europe/Amsterdam" in result
def test_no_timezone_when_specified(self):
"""Test timezone is not appended when already specified."""
query = "What time is it in UTC?"
memory_context = {
"profile": {"timezone": "Europe/Amsterdam"}
}
result = _build_enriched_query(query, memory_context)
assert result == query
def test_enrichment_adds_temperature_unit(self):
"""Test temperature unit is appended for weather queries."""
query = "What's the weather?"
memory_context = {
"profile": {"location": "Amsterdam"},
"preferences": {"temperature_unit": "celsius"}
}
result = _build_enriched_query(query, memory_context)
assert "temperature_unit=celsius" in result
def test_multiple_context_fields(self):
"""Test multiple context fields are appended."""
query = "What time and weather today?"
memory_context = {
"profile": {
"location": "Amsterdam",
"timezone": "Europe/Amsterdam"
},
"preferences": {"temperature_unit": "celsius"}
}
result = _build_enriched_query(query, memory_context)
assert "location=Amsterdam" in result
assert "timezone=Europe/Amsterdam" in result
assert "temperature_unit=celsius" in result
def test_no_enrichment_for_unrelated_query(self):
"""Test no enrichment for queries that don't need context."""
query = "Tell me a joke"
memory_context = {
"profile": {"location": "Amsterdam", "timezone": "Europe/Amsterdam"}
}
result = _build_enriched_query(query, memory_context)
assert result == query
+160
View File
@@ -8,9 +8,14 @@ import pytest
from unittest.mock import AsyncMock, patch, MagicMock from unittest.mock import AsyncMock, patch, MagicMock
from src.agents.delegation import ( from src.agents.delegation import (
ActionType,
DelegationTask, DelegationTask,
DelegationResult, DelegationResult,
HOUSEHOLD_THINK_MESSAGES,
STREAMING_DELEGATION_WRAPPERS,
delegate_to_librarian, delegate_to_librarian,
get_think_message,
_detect_action_type,
) )
@@ -193,3 +198,158 @@ class TestDelegateToLibrarian:
result = await delegate_to_librarian(task=original_task) result = await delegate_to_librarian(task=original_task)
assert result.task == original_task assert result.task == original_task
@pytest.mark.unit
class TestActionType:
"""Tests for the ActionType enum."""
def test_action_type_values(self):
"""Test ActionType enum values."""
assert ActionType.RETRIEVE.value == "retrieve"
assert ActionType.RESEARCH.value == "research"
assert ActionType.CREATE.value == "create"
assert ActionType.CONTROL.value == "control"
assert ActionType.RECORD.value == "record"
def test_action_type_is_enum(self):
"""Test ActionType is proper enum."""
assert len(ActionType) == 5
@pytest.mark.unit
class TestHouseholdThinkMessages:
"""Tests for HOUSEHOLD_THINK_MESSAGES mapping."""
def test_librarian_has_messages(self):
"""Test librarian has think messages."""
assert "librarian" in HOUSEHOLD_THINK_MESSAGES
assert ActionType.RETRIEVE in HOUSEHOLD_THINK_MESSAGES["librarian"]
assert ActionType.RESEARCH in HOUSEHOLD_THINK_MESSAGES["librarian"]
assert ActionType.CREATE in HOUSEHOLD_THINK_MESSAGES["librarian"]
def test_biographer_has_messages(self):
"""Test biographer has think messages."""
assert "biographer" in HOUSEHOLD_THINK_MESSAGES
assert ActionType.RETRIEVE in HOUSEHOLD_THINK_MESSAGES["biographer"]
assert ActionType.RECORD in HOUSEHOLD_THINK_MESSAGES["biographer"]
def test_housekeeper_has_messages(self):
"""Test housekeeper has think messages."""
assert "housekeeper" in HOUSEHOLD_THINK_MESSAGES
assert ActionType.RETRIEVE in HOUSEHOLD_THINK_MESSAGES["housekeeper"]
assert ActionType.CONTROL in HOUSEHOLD_THINK_MESSAGES["housekeeper"]
def test_messages_have_phases(self):
"""Test each action type has start/success/error messages."""
for expert, action_types in HOUSEHOLD_THINK_MESSAGES.items():
for action_type, messages in action_types.items():
assert "start" in messages, f"{expert}/{action_type} missing 'start'"
assert "success" in messages, f"{expert}/{action_type} missing 'success'"
assert "error" in messages, f"{expert}/{action_type} missing 'error'"
def test_messages_are_think_tags(self):
"""Test messages are wrapped in <think> tags."""
for expert, action_types in HOUSEHOLD_THINK_MESSAGES.items():
for action_type, messages in action_types.items():
for phase, msg in messages.items():
assert msg.startswith("<think>"), f"{expert}/{action_type}/{phase}"
assert msg.endswith("</think>"), f"{expert}/{action_type}/{phase}"
@pytest.mark.unit
class TestDetectActionType:
"""Tests for _detect_action_type function."""
def test_librarian_search_is_retrieve(self):
"""Test librarian search tasks are RETRIEVE."""
assert _detect_action_type("librarian", "search for Docker info") == ActionType.RETRIEVE
assert _detect_action_type("librarian", "find information about CI/CD") == ActionType.RETRIEVE
assert _detect_action_type("librarian", "look up Kubernetes docs") == ActionType.RETRIEVE
def test_librarian_web_search_is_research(self):
"""Test librarian web search tasks are RESEARCH."""
assert _detect_action_type("librarian", "search the web for news") == ActionType.RESEARCH
assert _detect_action_type("librarian", "find online resources") == ActionType.RESEARCH
assert _detect_action_type("librarian", "research internet sources") == ActionType.RESEARCH
def test_librarian_create_is_create(self):
"""Test librarian creation tasks are CREATE."""
assert _detect_action_type("librarian", "create a wiki page") == ActionType.CREATE
assert _detect_action_type("librarian", "write a new article") == ActionType.CREATE
assert _detect_action_type("librarian", "add a new entry") == ActionType.CREATE
def test_biographer_recall_is_retrieve(self):
"""Test biographer recall tasks are RETRIEVE."""
assert _detect_action_type("biographer", "what car do I drive?") == ActionType.RETRIEVE
assert _detect_action_type("biographer", "what is my job?") == ActionType.RETRIEVE
def test_biographer_record_is_record(self):
"""Test biographer record tasks are RECORD."""
assert _detect_action_type("biographer", "remember that I work at Acme") == ActionType.RECORD
assert _detect_action_type("biographer", "note that my car is a Tesla") == ActionType.RECORD
assert _detect_action_type("biographer", "save my preference for dark mode") == ActionType.RECORD
def test_housekeeper_status_is_retrieve(self):
"""Test housekeeper status tasks are RETRIEVE."""
assert _detect_action_type("housekeeper", "what devices are in the bedroom?") == ActionType.RETRIEVE
assert _detect_action_type("housekeeper", "is the living room light on?") == ActionType.RETRIEVE
def test_housekeeper_control_is_control(self):
"""Test housekeeper control tasks are CONTROL."""
assert _detect_action_type("housekeeper", "turn on the lights") == ActionType.CONTROL
assert _detect_action_type("housekeeper", "set brightness to 50%") == ActionType.CONTROL
assert _detect_action_type("housekeeper", "activate the movie scene") == ActionType.CONTROL
assert _detect_action_type("housekeeper", "toggle the fan") == ActionType.CONTROL
@pytest.mark.unit
class TestGetThinkMessage:
"""Tests for get_think_message function."""
def test_librarian_retrieve_start(self):
"""Test getting librarian retrieve start message."""
msg = get_think_message("librarian", "search for Docker", "start")
assert "<think>" in msg
assert "</think>" in msg
def test_librarian_create_success(self):
"""Test getting librarian create success message."""
msg = get_think_message("librarian", "create a wiki page", "success")
assert "<think>" in msg
assert "catalogued" in msg.lower()
def test_biographer_record_start(self):
"""Test getting biographer record start message."""
msg = get_think_message("biographer", "remember my preference", "start")
assert "<think>" in msg
assert "note" in msg.lower() or "biographer" in msg.lower()
def test_housekeeper_control_success(self):
"""Test getting housekeeper control success message."""
msg = get_think_message("housekeeper", "turn on the lights", "success")
assert "<think>" in msg
assert "configured" in msg.lower()
def test_unknown_expert_fallback(self):
"""Test unknown expert gets fallback message."""
msg = get_think_message("unknown_expert", "some task", "start")
assert "<think>" in msg
assert "unknown_expert" in msg.lower()
@pytest.mark.unit
class TestStreamingDelegationWrappers:
"""Tests for streaming delegation wrapper mapping."""
def test_streaming_wrappers_exist(self):
"""Test streaming wrappers mapping has all experts."""
assert "librarian" in STREAMING_DELEGATION_WRAPPERS
assert "biographer" in STREAMING_DELEGATION_WRAPPERS
assert "housekeeper" in STREAMING_DELEGATION_WRAPPERS
def test_streaming_wrappers_are_async_generators(self):
"""Test streaming wrappers are async generator functions."""
import inspect
for name, wrapper in STREAMING_DELEGATION_WRAPPERS.items():
assert inspect.isasyncgenfunction(wrapper), f"{name} is not an async generator"