tatlock/IMPLEMENTATION_ROADMAP.md

Tatlock Implementation Roadmap

Reference: See PHILOSOPHY.md for the target architecture and vision

This document outlines the phased implementation plan to transform the current OpenAI-compatible API into the full Tatlock household butler system.

Current State (v1.2.0 - Phase F Complete)

What we have:

• ✅ The Orchestrator - FastAPI infrastructure layer
  • OpenAI-compatible API endpoints (Responses API + Chat Completions)
  • Streaming coordination and conversation management
  • Response format with reasoning support
  • Test infrastructure (~400 tests)
• ✅ Two-Tier Architecture
  • The Steward analyzes requests and recommends capabilities
  • Tatlock coordinates execution with scoped tools
  • Real-time streaming of analysis and reasoning
• ✅ Household Staff
  • Tatlock (Butler): Primary interface with witty personality
  • The Steward: Request analysis and capability recommendation
  • The Librarian: Research via library-desk HybridRAG + wiki
  • The Biographer: User memory, profiles, preferences, semantic recall
• ✅ Core Tools
  • Calculator, Date/Time toolkit, Web search (SearXNG)
• ✅ Memory System
  • Direct access layer (memory_service) for fast lookups
  • Vector storage (Qdrant) for semantic recall
  • Session cache (Redis) with 24h TTL
  • Multi-tenancy via ContextVar
• ✅ Mock agent (lorem-tester for testing)

What we need:

• More household staff (Developer, Secretary, Handyman, Housekeeper)
• MCP (Model Context Protocol) integration
• Dynamic model switching for specialized tasks
• Full multi-tenant database (PostgreSQL)

---

Phase 1: Real LLM Integration - PydanticAI + Tools

Goal: Connect to actual language models and establish the base plumbing

Note: Ollama is an external service dependency (already running separately)

Deliverables

1. PydanticAI Integration ✅

   • PydanticAI → Ollama connection ✅
   • Agent creation patterns ✅
   • Streaming response handling ✅
   • Error handling and retries ✅

2. Convert Tatlock Agent ✅

   • Convert Tatlock agent from mock to PydanticAI ✅
   • British butler personality prompt ✅
   • Research-oriented mindset ✅
   • Streaming to reasoning output ✅
   • Tool calling framework setup ✅

3. Permanent Tools ✅

   • Calculator: Safe mathematical expression evaluation ✅
   • Date/Time toolkit: Current time, relative dates, time differences ✅
   • Web search: SearXNG integration (external service) ✅
   • Tool registration with PydanticAI ✅

4. Testing Infrastructure ✅

   • Integration tests with real LLM ✅
   • Tool functionality tests ✅
   • Response quality validation ✅
   • 131 tests, 81.78% coverage ✅

Success Criteria

• PydanticAI agents can call Ollama (mistral-nemo:latest)
• Streaming works end-to-end
• Tool calling framework functional
• Permanent tools working (calculator, date/time, search)
• Tests pass with real LLM
• Can switch models dynamically (e.g., Codestral for code)

Status

✅ MOSTLY COMPLETE - Tatlock agent functional with permanent tools

Remaining Work

• Dynamic model switching for specialized tasks (e.g., Codestral for coding)

Why First?

Without real LLM integration, we can't meaningfully implement the Steward/Butler pattern. Everything else depends on having actual AI agents working.

---

Phase 2: Orchestration Layer - The Steward

Goal: Implement the first-tier LLM call for tool/agent selection

Purpose: The Steward performs crucial preparatory work before Tatlock engages with a request. By analyzing incoming requests and determining which tools, services, and household staff members will be needed, the Steward creates a curated recommendation that streamlines Tatlock's work and prevents cognitive overload.

Core Architecture

The Steward operates as the first tier in the two-tier request flow:

User Request → Orchestrator → Steward Analysis → Recommendations → Tatlock (with scoped tools/agents)

Key Principle: The Steward narrows the scope to only relevant capabilities, making Tatlock's decision-making cleaner and more focused.

Deliverables

1. Tool & Agent Registry System

Purpose: Centralized catalog of all available capabilities for the Steward to recommend

Implementation Details:

• Registry Module (src/core/registry.py)

  • Tool registration decorator pattern
  • Agent registration with capability metadata
  • Category-based organization (computation, information, automation, communication)
  • Dynamic tool/agent discovery and loading

• Tool Metadata Schema

  {
    "name": "calculator",
    "category": "computation",
    "description": "Safe mathematical expression evaluation",
    "capabilities": ["arithmetic", "algebra", "trigonometry"],
    "cost": "low",  # computational cost indicator
    "requires_network": false
  }
  

• Agent Metadata Schema

  {
    "name": "developer",
    "role": "The Developer",
    "category": "technical",
    "description": "Software development assistance",
    "domains": ["code_generation", "debugging", "architecture"],
    "specialized_model": "codestral",  # optional
    "cost": "high"
  }
  

• Registry API

  • get_all_tools() - List all available tools
  • get_all_agents() - List all expert agents
  • get_by_category(category) - Filter by category
  • search_by_capability(query) - Semantic search (future: vector search)

Testing:

• Unit tests for registration and retrieval
• Test dynamic loading of new tools/agents
• Validate metadata schemas

2. Steward PydanticAI Agent

Purpose: First-tier LLM that analyzes requests and recommends relevant tools/agents

Implementation Details:

• Agent Module (src/agents/steward.py)

  from pydantic_ai import Agent, RunContext
  from pydantic import BaseModel
  
  class StewardRecommendation(BaseModel):
      """Structured output from Steward analysis"""
      recommended_tools: list[str]
      recommended_agents: list[str]
      reasoning: str
      estimated_complexity: str  # "simple", "moderate", "complex"
      requires_multi_step: bool
  
  steward = Agent(
      'ollama:mistral-nemo',  # Same base model as Tatlock
      result_type=StewardRecommendation,
      system_prompt="""..."""
  )
  

• System Prompt Engineering

  • Role: Estate steward responsible for efficient household coordination
  • Task: Analyze requests to determine needed resources
  • Output: Structured recommendations with reasoning
  • Constraints: Be conservative (recommend only truly relevant capabilities)
  • Context: Full registry of available tools and agents

• Steward Tools

  @steward.tool
  def get_available_capabilities(ctx: RunContext) -> dict:
      """Get catalog of all available tools and agents."""
      return {
          "tools": registry.get_all_tools(),
          "agents": registry.get_all_agents()
      }
  

• Request Analysis Flow

  1. Receive user request
  2. Query capability registry via tool
  3. Analyze request for required capabilities
  4. Generate structured recommendation
  5. Format as note to Tatlock

Testing:

• Test various request types (simple, complex, multi-domain)
• Verify recommendations are relevant and not over-inclusive
• Test structured output parsing
• Validate reasoning quality

3. Request Preprocessing Pipeline

Purpose: Integration layer that routes requests through Steward before Tatlock

Implementation Details:

• Preprocessing Module (src/core/preprocessing.py)

  async def preprocess_request(user_request: str) -> EnrichedRequest:
      """
      1. Call Steward for analysis
      2. Get recommendations
      3. Enrich original request
      4. Return scoped context for Tatlock
      """
      # Get Steward analysis
      steward_result = await steward.run(user_request)
      recommendations = steward_result.data
  
      # Create note to Tatlock
      steward_note = format_steward_note(recommendations)
  
      # Build scoped tool/agent list
      scoped_tools = get_scoped_tools(recommendations.recommended_tools)
      scoped_agents = get_scoped_agents(recommendations.recommended_agents)
  
      return EnrichedRequest(
          original_request=user_request,
          steward_note=steward_note,
          available_tools=scoped_tools,
          available_agents=scoped_agents,
          metadata=recommendations
      )
  

• Note Formatting

  === Internal Note from the Steward ===
  
  Request Analysis:
  {steward reasoning}
  
  Recommended Tools:
  - calculator: For mathematical computations
  - web_search: To find current information
  
  Recommended Household Staff:
  - The Developer: For code generation assistance
  
  Estimated Complexity: moderate
  ===================================
  
  [Original User Request]
  

• Orchestrator Integration

  • Modify src/responses/service.py to call preprocessing
  • Prepend Steward note to request before sending to Tatlock
  • Limit Tatlock's tool access to recommended tools only
  • Stream Steward's reasoning to output

Testing:

• Integration tests for full preprocessing flow
• Test request enrichment format
• Verify tool scoping works correctly
• Test streaming of Steward reasoning

4. Real-Time Transparency

Purpose: Stream Steward's analysis to user's reasoning output

Implementation Details:

• Streaming Integration (src/responses/streaming.py)

  • Add Steward analysis phase to stream
  • Format as reasoning item
  • Include recommendation summary

• Example Output to User:

  [Reasoning]
  Consulting the Steward for resource planning...
  
  The Steward's Analysis:
  - Request requires mathematical computation
  - Need to verify current information via web search
  - May benefit from Developer's code expertise
  
  Recommended: calculator, web_search, The Developer
  
  Proceeding with scoped resources...
  

Testing:

• Test streaming of Steward analysis
• Verify formatting in Open WebUI
• Test error handling if Steward fails

5. Model Efficiency Optimization

Purpose: Ensure the base model stays loaded in VRAM

Implementation Details:

• Shared Model Configuration

  • Both Steward and Tatlock use ollama:mistral-nemo by default
  • Sequential calls (Steward → Tatlock) keep model hot
  • No reload delays between tiers

• Performance Monitoring

  • Log response times for Steward calls
  • Track total request latency (Steward + Tatlock)
  • Identify optimization opportunities

Testing:

• Benchmark Steward → Tatlock call latency
• Verify model stays loaded between calls
• Test performance under load

Implementation Strategy

Week 1-2: Foundation

• Design and implement registry system
• Create tool/agent metadata schemas
• Build registry API with tests
• Migrate existing tools to registry

Week 3-4: Steward Agent

• Create Steward PydanticAI agent
• Engineer system prompt for analysis
• Implement structured recommendation output
• Add registry query tool
• Test with various request types

Week 5-6: Integration

• Build request preprocessing pipeline
• Implement note formatting
• Integrate with Orchestrator
• Add streaming transparency
• Tool scoping for Tatlock

Week 7: Testing & Refinement

• End-to-end integration tests
• Performance optimization
• Prompt refinement based on results
• Documentation and examples

Success Criteria

• Steward analyzes incoming requests using PydanticAI agent
• Produces structured recommendations (tools, agents, reasoning)
• Recommendations formatted as prepended note to Tatlock
• Tool registry is queryable and extensible via clean API
• Steward output visible in reasoning stream for transparency
• Only recommended tools available to Tatlock (scoped context)
• Base model stays loaded between Steward and Tatlock calls
• Recommendations are accurate (not over/under-inclusive)
• Integration tests pass for full Steward → Tatlock flow

Status

✅ COMPLETE (v0.2.5)

Performance Targets

• Steward Analysis Time: < 2 seconds for typical requests
• Total Added Latency: < 3 seconds including streaming
• Recommendation Accuracy: > 90% relevance (manual evaluation)
• Model Reload Delay: 0 seconds (model stays hot)

Risk Mitigation

Risk: Steward recommendations too broad (defeats purpose)

• Mitigation: Conservative prompt engineering, test with diverse requests, iterate

Risk: Added latency unacceptable to users

• Mitigation: Stream Steward reasoning for transparency, optimize prompt, parallel processing where possible

Risk: Tool registry becomes unwieldy

• Mitigation: Good categorization, semantic search (future), regular pruning

Risk: Steward and Tatlock models compete for VRAM

• Mitigation: Use same base model, sequential calls, monitor memory

Future Enhancements (Post-Phase 2)

• Semantic Search: Vector-based capability search instead of metadata lookup
• Learning from Usage: Track which recommendations work well, adjust over time
• Confidence Scores: Steward provides confidence for each recommendation
• Request Classification: Cache classifications for similar requests
• Multi-Model Support: Allow Steward to recommend specialized models for specific tasks

Estimated Effort

7-8 weeks - Core intelligence routing with comprehensive implementation

Why Second?

The Steward is the foundation of the household architecture. Without it, we'd need to expose all tools/agents to Tatlock, creating cognitive overload and poor decision-making. The Steward enables the focused expertise pattern that makes the whole system work.

---

Phase 3: The Butler - Tatlock Agent

Goal: Implement the second-tier coordinator with personality within the existing Orchestrator infrastructure

Context: The Orchestrator (FastAPI infrastructure) already exists. This phase implements the real Tatlock PydanticAI agent to replace the current mock agent.

Deliverables

1. Butler Agent (Tatlock)

   • PydanticAI agent implementation within Orchestrator
   • Personality prompt engineering (witty British butler)
   • Tool calling framework
   • Multi-agent coordination logic

2. Scoped Tool Access

   • Filter tools based on Steward recommendations
   • Dynamic tool loading for Butler context
   • Tool execution framework
   • Result aggregation

3. Real-Time Reasoning Output

   • Stream all Butler activities to reasoning output
   • Tool call progress indicators
   • Expert agent consultation messages
   • Wait time transparency

Success Criteria

• Tatlock receives enriched requests (user + Steward notes)
• Only recommended tools are available
• Tatlock coordinates multiple tool calls
• All actions streamed to reasoning output
• Responses have consistent personality
• Synthesizes multi-source results coherently

Status

✅ COMPLETE (v1.1.0)

Estimated Effort

4-5 weeks - Complex coordination logic

---

Phase 4: Expert Household Staff - Core Agents

Goal: Implement the initial set of domain-specific expert agents

Priority Expert Agents

1. The Librarian (Research & Knowledge Management) ✅ COMPLETE (v1.1.0)

   • Research assistance via library-desk HybridRAG
   • Wiki page management (search, create, update)
   • Semantic vector search
   • Knowledge graph queries
   • Dossier browsing

2. The Biographer (User Memory) ✅ COMPLETE (v1.2.0)

   • User profile management (name, location, timezone)
   • Preference storage (units, theme)
   • Semantic memory recall ("What car do I drive?")
   • Fact storage from conversations
   • Session context caching

3. The Developer (Software Development) 🔜 Planned

   • Code generation assistance
   • Debugging support
   • Documentation generation
   • Architecture guidance
   • Rationale: Directly supports building the system itself

4. The Handyman (System Maintenance) 🔜 Planned

   • System status queries
   • Log analysis
   • Basic troubleshooting
   • Infrastructure monitoring

5. The Secretary (Scheduling & Organization) 🔜 Planned

   • Calendar integration
   • Task management
   • Reminder system
   • Schedule conflict detection

6. The Housekeeper (Home Automation) 🔜 Planned

   • Home Assistant integration
   • Device control interface
   • Status queries
   • Automation triggers

Each Agent Includes

• Specialized prompt and personality
• Domain-specific tools
• MCP integration points (where applicable)
• Integration with Butler orchestration

Success Criteria

• Each agent implemented as separate module
• Agents callable via tool framework
• Agents use specialized prompts
• Results integrate cleanly with Butler
• Can invoke specialized models (e.g., Codestral for Developer)

Status

🔶 PARTIAL - Librarian and Biographer complete, others planned

Estimated Effort

6-8 weeks - Parallel development possible

---

Phase 5: Persistence Layer - Database & Multi-Tenancy

Goal: Add persistent storage and multi-user support when needed

Deliverables

1. PostgreSQL Integration

   • Docker compose configuration for PostgreSQL
   • Database schema design with tenant isolation
   • Alembic migrations setup
   • SQLAlchemy models

2. Multi-Tenant Architecture

   • Tenant identification middleware
   • Tenant-scoped database sessions
   • User authentication system (basic)
   • Per-tenant data isolation

3. Core Data Models

   • Users and tenants
   • Conversations and messages (migrate from in-memory)
   • Agent interactions log
   • System configuration and preferences

4. Migration Strategy

   • Gradual migration from in-memory to database
   • Backward compatibility during transition
   • Data export/import utilities

Success Criteria

• PostgreSQL container running
• Multiple users can authenticate separately
• Each user sees only their own data
• Conversations persist across restarts
• Database migrations work correctly
• Tests verify tenant isolation

Estimated Effort

3-4 weeks - Data layer foundation

Why Later?

The core orchestration (Steward → Butler → Experts) can work entirely with in-memory state. We only need database persistence when we want conversations to survive restarts and multiple users to have isolated experiences.

---

Phase 6: Extended Services Integration

Goal: Connect to additional supporting services

Services to Integrate

1. Redis (Memory & Caching) ✅ COMPLETE (v1.2.0)

   • Benchmark storage (db=1)
   • Memory cache for sessions (db=2)
   • 24h TTL for session context
   • Recent entities tracking

2. Qdrant (Vector Storage) ✅ COMPLETE (v1.2.0)

   • Per-user memory collections
   • 768-dim nomic-embed-text vectors
   • Semantic search for recall
   • Type-based filtering

3. SearxNG (Web Search) ✅ COMPLETE (v0.2.0)

   • Search tool integration
   • Result processing
   • Privacy-preserving queries

4. library-desk (Research API) ✅ COMPLETE (v1.1.0)

   • HybridRAG search
   • Wiki management
   • Knowledge graph queries

Success Criteria

• Services communicate correctly
• Tatlock can invoke web search
• Redis used for session data
• Qdrant stores user memories
• Ollama serves the base model

Status

✅ COMPLETE - All core services integrated

Estimated Effort

3-4 weeks - Infrastructure setup

---

Phase 7: MCP (Model Context Protocol) Integration

Goal: Enable rich tool integrations via MCP

Deliverables

1. MCP Server Framework

   • MCP server implementation
   • Tool registration via MCP
   • Schema validation
   • Error handling

2. MCP Client in Agents

   • PydanticAI MCP integration
   • Tool discovery from MCP servers
   • Dynamic tool loading
   • Result processing

3. Initial MCP Tools

   • File system operations
   • Database queries
   • API integrations
   • System commands

Success Criteria

• MCP server running
• Tools exposed via MCP protocol
• Agents can discover and use MCP tools
• New tools addable without code changes
• MCP tools visible in Steward recommendations

Estimated Effort

3-4 weeks - Standards-based integration

---

Phase 8: Advanced Memory & Context

Goal: Implement sophisticated memory and context management

Deliverables

1. Long-Term Memory ✅ COMPLETE (v1.2.0 - Phase F)

   • Memory service for direct key-based access
   • Qdrant vector storage for semantic recall
   • Embedding via nomic-embed-text
   • The Biographer agent for memory management

2. Session Memory ✅ COMPLETE (v1.2.0)

   • Redis session cache with 24h TTL
   • Recent entities tracking
   • Conversation context preservation
   • Multi-tenancy via ContextVar

3. Steward Integration ✅ COMPLETE (v1.2.0)

   • Memory pre-fetch during request analysis
   • Profile/preferences included in context
   • Keyword-based context determination

4. Context Management 🔜 Future

   • Smart context window trimming
   • Conversation branching
   • Topic tracking
   • Memory retrieval integration

5. Personalization 🔜 Future

   • User preference learning
   • Interaction pattern analysis
   • Adaptive responses
   • Custom agent personalities per user

Success Criteria

• User facts stored in Qdrant with semantic search
• Profile and preferences accessible via memory_service
• Session context cached in Redis
• User preferences affect responses (via Steward pre-fetch)
• Conversations automatically embedded to Qdrant
• Memory improves over time (learning from interactions)

Status

🔶 PARTIAL - Core memory system complete, advanced features planned

Estimated Effort

4-5 weeks - AI/ML heavy (remaining work)

---

Phase 9: Extended Household Staff

Goal: Add specialized agents for additional domains

Future Agents

1. The Librarian (Knowledge Management)

   • Personal documentation indexing
   • Research assistance
   • Knowledge base queries
   • Reference management

2. The Accountant (Financial Tracking)

   • Expense tracking
   • Budget monitoring
   • Financial reports
   • Transaction categorization

3. The Chef (Meal Planning)

   • Recipe management
   • Meal planning
   • Nutrition tracking
   • Grocery lists

4. Others as Needed

   • Domain-specific as requirements emerge

Success Criteria

• Each new agent follows household pattern
• Integrates with Steward/Butler flow
• Has appropriate specialized tools
• Documented in PHILOSOPHY.md updates

Estimated Effort

Ongoing - Add as needed

---

Phase 10: User Experience Refinement

Goal: Polish the interaction experience

Deliverables

1. Personality Tuning

   • Refine Tatlock's wit and tone
   • Consistent household character
   • Cultural references appropriate
   • Humor that doesn't annoy

2. Transparency Improvements

   • Better progress indicators
   • Clearer reasoning explanations
   • Informative wait messages
   • Error message clarity

3. Performance Optimization

   • Response time improvements
   • Model loading optimization
   • Caching strategies
   • Streaming smoothness

Success Criteria

• Users find Tatlock engaging
• Wait times feel reasonable
• Errors are understandable
• System feels responsive

Estimated Effort

Ongoing - Continuous improvement

---

Phase 11: Production Hardening

Goal: Make the system production-ready for homelab deployment

Deliverables

1. Deployment

   • Complete docker-compose stack
   • Environment configuration
   • Backup strategies
   • Update procedures

2. Monitoring

   • Health checks
   • Performance metrics
   • Error tracking
   • Usage analytics

3. Security

   • Authentication hardening
   • Rate limiting
   • Input validation
   • Audit logging

4. Documentation

   • Installation guide
   • Configuration reference
   • Troubleshooting guide
   • Architecture documentation

Success Criteria

• One-command deployment
• System health is monitorable
• Secure for homelab use
• Well documented

Estimated Effort

3-4 weeks - Production polish

---

Dependencies Between Phases

Phase 1 (Ollama + PydanticAI) ← Foundation for all AI
    ↓
Phase 2 (Steward)
    ↓
Phase 3 (Butler/Tatlock)
    ↓
Phase 4 (Expert Agents) ← Phase 7 (MCP) can enhance
    ↓
Phase 5 (Database/Multi-Tenancy) ← Can be deferred
    ↓
Phase 6 (Extended Services) → Phase 8 (Advanced Memory)
    ↓
Phase 9 (Extended Staff) → Phase 10 (UX) → Phase 11 (Production)

Critical Path: Phases 1 → 2 → 3 → 4 must be sequential Can Be Deferred: Phase 5 (Database) until you need persistence Parallel Opportunities: Phase 6 and 7 can overlap; Phase 9 and 10 ongoing

---

Overall Timeline Estimate

Minimum Viable Household (Phases 1-4): 15-20 weeks

• Working Steward → Butler → Expert Agents with real LLM
• In-memory state (no persistence needed yet)
• Core household functional

With Persistence (Phases 1-5): 18-24 weeks

• Add database and multi-tenancy
• Conversations survive restarts
• Multiple users supported

Full-Featured System (Phases 1-9): 35-45 weeks

• All services integrated
• Advanced memory and context
• Extended household staff

Production-Ready (All phases): 40-50 weeks

• Polished UX
• Hardened for homelab deployment
• Fully documented

Note: Timeline assumes consistent part-time development effort

---

Success Metrics

Technical

• System implements PHILOSOPHY.md patterns
• All household roles functional
• Multi-tenant isolation verified
• Real-time reasoning transparency working
• MCP integration complete

User Experience

• Tatlock feels like interacting with a butler
• Wait times are transparent and acceptable
• Expert agents provide value in their domains
• System is reliable and trustworthy

Architecture

• Clean separation between household roles
• Easy to add new agents/tools
• Model efficiency (base model stays loaded)
• Scales to household + friends usage

---

Risk Management

High Risk Items

1. PydanticAI + Ollama integration complexity

   • Mitigation: Prototype early, iterate on connection layer

2. Multi-agent coordination complexity

   • Mitigation: Start simple, add coordination gradually

3. Model performance on homelab hardware

   • Mitigation: Model selection, quantization, optimization

4. Prompt engineering for personality consistency

   • Mitigation: Extensive testing, user feedback, iteration

Medium Risk Items

• MCP protocol adoption and tooling maturity
• Vector embedding quality for memory
• Home automation integration variability
• User authentication security

---

Next Steps

1. Priority: Implement The Developer agent for code assistance
2. Integration: Add Home Assistant integration for The Housekeeper
3. Calendar: Integrate scheduling service for The Secretary
4. Ongoing: Add more household staff as needed

---

Document Status: Active planning document Created: 2025-12-06 Last Updated: 2025-12-13