tatlock/IMPLEMENTATION_ROADMAP.md

# Tatlock Implementation Roadmap

> **Reference**: See [PHILOSOPHY.md](PHILOSOPHY.md) for the target architecture and vision

This document outlines the phased implementation plan to transform the current OpenAI-compatible API into the full Tatlock household butler system.

## Current State (v0.1.1+ - Phase 1 Mostly Complete)

**What we have**:
- ✅ **The Orchestrator** - FastAPI infrastructure layer
  - OpenAI-compatible API endpoints (Responses API + Chat Completions)
  - Streaming coordination and conversation management
  - Response format with reasoning support
  - Test infrastructure (131 tests, 81.78% coverage)
- ✅ **Tatlock Agent** - Real PydanticAI integration
  - Connected to Ollama (mistral-nemo:latest)
  - British butler personality with research mindset
  - Streaming responses with reasoning
  - Tool calling framework functional
- ✅ **Permanent Tools**
  - Calculator (safe mathematical expressions)
  - Date/Time toolkit (current time, relative dates, time differences)
  - Web search (SearXNG integration)
- ✅ Mock agent (lorem-tester for testing)
- ✅ Agent interface abstraction

**What we need**:
- **The Household** - Full multi-agent coordination:
  - The Steward (first-tier request analysis)
  - Tatlock coordination layer (expert agent delegation)
  - Expert household staff agents (Librarian, Developer, Handyman, etc.)
- Multi-tenant database architecture
- Containerized service ecosystem
- MCP (Model Context Protocol) integration
- Dynamic model switching for specialized tasks

---

## Phase 1: Real LLM Integration - PydanticAI + Tools

**Goal**: Connect to actual language models and establish the base plumbing

**Note**: Ollama is an external service dependency (already running separately)

### Deliverables

1. **PydanticAI Integration** ✅
   - PydanticAI → Ollama connection ✅
   - Agent creation patterns ✅
   - Streaming response handling ✅
   - Error handling and retries ✅

2. **Convert Tatlock Agent** ✅
   - Convert Tatlock agent from mock to PydanticAI ✅
   - British butler personality prompt ✅
   - Research-oriented mindset ✅
   - Streaming to reasoning output ✅
   - Tool calling framework setup ✅

3. **Permanent Tools** ✅
   - Calculator: Safe mathematical expression evaluation ✅
   - Date/Time toolkit: Current time, relative dates, time differences ✅
   - Web search: SearXNG integration (external service) ✅
   - Tool registration with PydanticAI ✅

4. **Testing Infrastructure** ✅
   - Integration tests with real LLM ✅
   - Tool functionality tests ✅
   - Response quality validation ✅
   - 131 tests, 81.78% coverage ✅

### Success Criteria
- [x] **PydanticAI agents can call Ollama** (mistral-nemo:latest)
- [x] **Streaming works end-to-end**
- [x] **Tool calling framework functional**
- [x] **Permanent tools working** (calculator, date/time, search)
- [x] **Tests pass with real LLM**
- [ ] Can switch models dynamically (e.g., Codestral for code)

### Status
**✅ MOSTLY COMPLETE** - Tatlock agent functional with permanent tools

### Remaining Work
- Dynamic model switching for specialized tasks (e.g., Codestral for coding)

### Why First?
Without real LLM integration, we can't meaningfully implement the Steward/Butler pattern. Everything else depends on having actual AI agents working.

---

## Phase 2: Orchestration Layer - The Steward

**Goal**: Implement the first-tier LLM call for tool/agent selection

**Purpose**: The Steward performs crucial preparatory work before Tatlock engages with a request. By analyzing incoming requests and determining which tools, services, and household staff members will be needed, the Steward creates a curated recommendation that streamlines Tatlock's work and prevents cognitive overload.

### Core Architecture

The Steward operates as the first tier in the two-tier request flow:

```
User Request → Orchestrator → Steward Analysis → Recommendations → Tatlock (with scoped tools/agents)
```

**Key Principle**: The Steward narrows the scope to only relevant capabilities, making Tatlock's decision-making cleaner and more focused.

### Deliverables

#### 1. Tool & Agent Registry System

**Purpose**: Centralized catalog of all available capabilities for the Steward to recommend

**Implementation Details**:
- **Registry Module** (`src/core/registry.py`)
  - Tool registration decorator pattern
  - Agent registration with capability metadata
  - Category-based organization (computation, information, automation, communication)
  - Dynamic tool/agent discovery and loading

- **Tool Metadata Schema**
  ```python
  {
    "name": "calculator",
    "category": "computation",
    "description": "Safe mathematical expression evaluation",
    "capabilities": ["arithmetic", "algebra", "trigonometry"],
    "cost": "low",  # computational cost indicator
    "requires_network": false
  }
  ```

- **Agent Metadata Schema**
  ```python
  {
    "name": "developer",
    "role": "The Developer",
    "category": "technical",
    "description": "Software development assistance",
    "domains": ["code_generation", "debugging", "architecture"],
    "specialized_model": "codestral",  # optional
    "cost": "high"
  }
  ```

- **Registry API**
  - `get_all_tools()` - List all available tools
  - `get_all_agents()` - List all expert agents
  - `get_by_category(category)` - Filter by category
  - `search_by_capability(query)` - Semantic search (future: vector search)

**Testing**:
- Unit tests for registration and retrieval
- Test dynamic loading of new tools/agents
- Validate metadata schemas

#### 2. Steward PydanticAI Agent

**Purpose**: First-tier LLM that analyzes requests and recommends relevant tools/agents

**Implementation Details**:

- **Agent Module** (`src/agents/steward.py`)
  ```python
  from pydantic_ai import Agent, RunContext
  from pydantic import BaseModel

  class StewardRecommendation(BaseModel):
      """Structured output from Steward analysis"""
      recommended_tools: list[str]
      recommended_agents: list[str]
      reasoning: str
      estimated_complexity: str  # "simple", "moderate", "complex"
      requires_multi_step: bool

  steward = Agent(
      'ollama:mistral-nemo',  # Same base model as Tatlock
      result_type=StewardRecommendation,
      system_prompt="""..."""
  )
  ```

- **System Prompt Engineering**
  - Role: Estate steward responsible for efficient household coordination
  - Task: Analyze requests to determine needed resources
  - Output: Structured recommendations with reasoning
  - Constraints: Be conservative (recommend only truly relevant capabilities)
  - Context: Full registry of available tools and agents

- **Steward Tools**
  ```python
  @steward.tool
  def get_available_capabilities(ctx: RunContext) -> dict:
      """Get catalog of all available tools and agents."""
      return {
          "tools": registry.get_all_tools(),
          "agents": registry.get_all_agents()
      }
  ```

- **Request Analysis Flow**
  1. Receive user request
  2. Query capability registry via tool
  3. Analyze request for required capabilities
  4. Generate structured recommendation
  5. Format as note to Tatlock

**Testing**:
- Test various request types (simple, complex, multi-domain)
- Verify recommendations are relevant and not over-inclusive
- Test structured output parsing
- Validate reasoning quality

#### 3. Request Preprocessing Pipeline

**Purpose**: Integration layer that routes requests through Steward before Tatlock

**Implementation Details**:

- **Preprocessing Module** (`src/core/preprocessing.py`)
  ```python
  async def preprocess_request(user_request: str) -> EnrichedRequest:
      """
      1. Call Steward for analysis
      2. Get recommendations
      3. Enrich original request
      4. Return scoped context for Tatlock
      """
      # Get Steward analysis
      steward_result = await steward.run(user_request)
      recommendations = steward_result.data

      # Create note to Tatlock
      steward_note = format_steward_note(recommendations)

      # Build scoped tool/agent list
      scoped_tools = get_scoped_tools(recommendations.recommended_tools)
      scoped_agents = get_scoped_agents(recommendations.recommended_agents)

      return EnrichedRequest(
          original_request=user_request,
          steward_note=steward_note,
          available_tools=scoped_tools,
          available_agents=scoped_agents,
          metadata=recommendations
      )
  ```

- **Note Formatting**
  ```
  === Internal Note from the Steward ===

  Request Analysis:
  {steward reasoning}

  Recommended Tools:
  - calculator: For mathematical computations
  - web_search: To find current information

  Recommended Household Staff:
  - The Developer: For code generation assistance

  Estimated Complexity: moderate
  ===================================

  [Original User Request]
  ```

- **Orchestrator Integration**
  - Modify `src/responses/service.py` to call preprocessing
  - Prepend Steward note to request before sending to Tatlock
  - Limit Tatlock's tool access to recommended tools only
  - Stream Steward's reasoning to output

**Testing**:
- Integration tests for full preprocessing flow
- Test request enrichment format
- Verify tool scoping works correctly
- Test streaming of Steward reasoning

#### 4. Real-Time Transparency

**Purpose**: Stream Steward's analysis to user's reasoning output

**Implementation Details**:

- **Streaming Integration** (`src/responses/streaming.py`)
  - Add Steward analysis phase to stream
  - Format as reasoning item
  - Include recommendation summary

- **Example Output to User**:
  ```
  [Reasoning]
  Consulting the Steward for resource planning...

  The Steward's Analysis:
  - Request requires mathematical computation
  - Need to verify current information via web search
  - May benefit from Developer's code expertise

  Recommended: calculator, web_search, The Developer

  Proceeding with scoped resources...
  ```

**Testing**:
- Test streaming of Steward analysis
- Verify formatting in Open WebUI
- Test error handling if Steward fails

#### 5. Model Efficiency Optimization

**Purpose**: Ensure the base model stays loaded in VRAM

**Implementation Details**:

- **Shared Model Configuration**
  - Both Steward and Tatlock use `ollama:mistral-nemo` by default
  - Sequential calls (Steward → Tatlock) keep model hot
  - No reload delays between tiers

- **Performance Monitoring**
  - Log response times for Steward calls
  - Track total request latency (Steward + Tatlock)
  - Identify optimization opportunities

**Testing**:
- Benchmark Steward → Tatlock call latency
- Verify model stays loaded between calls
- Test performance under load

### Implementation Strategy

#### Week 1-2: Foundation
- [ ] Design and implement registry system
- [ ] Create tool/agent metadata schemas
- [ ] Build registry API with tests
- [ ] Migrate existing tools to registry

#### Week 3-4: Steward Agent
- [ ] Create Steward PydanticAI agent
- [ ] Engineer system prompt for analysis
- [ ] Implement structured recommendation output
- [ ] Add registry query tool
- [ ] Test with various request types

#### Week 5-6: Integration
- [ ] Build request preprocessing pipeline
- [ ] Implement note formatting
- [ ] Integrate with Orchestrator
- [ ] Add streaming transparency
- [ ] Tool scoping for Tatlock

#### Week 7: Testing & Refinement
- [ ] End-to-end integration tests
- [ ] Performance optimization
- [ ] Prompt refinement based on results
- [ ] Documentation and examples

### Success Criteria

- [ ] **Steward analyzes incoming requests** using PydanticAI agent
- [ ] **Produces structured recommendations** (tools, agents, reasoning)
- [ ] **Recommendations formatted as prepended note** to Tatlock
- [ ] **Tool registry is queryable and extensible** via clean API
- [ ] **Steward output visible in reasoning stream** for transparency
- [ ] **Only recommended tools available** to Tatlock (scoped context)
- [ ] **Base model stays loaded** between Steward and Tatlock calls
- [ ] **Recommendations are accurate** (not over/under-inclusive)
- [ ] **Integration tests pass** for full Steward → Tatlock flow

### Performance Targets

- **Steward Analysis Time**: < 2 seconds for typical requests
- **Total Added Latency**: < 3 seconds including streaming
- **Recommendation Accuracy**: > 90% relevance (manual evaluation)
- **Model Reload Delay**: 0 seconds (model stays hot)

### Risk Mitigation

**Risk**: Steward recommendations too broad (defeats purpose)
- Mitigation: Conservative prompt engineering, test with diverse requests, iterate

**Risk**: Added latency unacceptable to users
- Mitigation: Stream Steward reasoning for transparency, optimize prompt, parallel processing where possible

**Risk**: Tool registry becomes unwieldy
- Mitigation: Good categorization, semantic search (future), regular pruning

**Risk**: Steward and Tatlock models compete for VRAM
- Mitigation: Use same base model, sequential calls, monitor memory

### Future Enhancements (Post-Phase 2)

- **Semantic Search**: Vector-based capability search instead of metadata lookup
- **Learning from Usage**: Track which recommendations work well, adjust over time
- **Confidence Scores**: Steward provides confidence for each recommendation
- **Request Classification**: Cache classifications for similar requests
- **Multi-Model Support**: Allow Steward to recommend specialized models for specific tasks

### Estimated Effort

**7-8 weeks** - Core intelligence routing with comprehensive implementation

### Why Second?

The Steward is the foundation of the household architecture. Without it, we'd need to expose all tools/agents to Tatlock, creating cognitive overload and poor decision-making. The Steward enables the focused expertise pattern that makes the whole system work.

---

## Phase 3: The Butler - Tatlock Agent

**Goal**: Implement the second-tier coordinator with personality within the existing Orchestrator infrastructure

**Context**: The Orchestrator (FastAPI infrastructure) already exists. This phase implements the real Tatlock PydanticAI agent to replace the current mock agent.

### Deliverables

1. **Butler Agent (Tatlock)**
   - PydanticAI agent implementation within Orchestrator
   - Personality prompt engineering (witty British butler)
   - Tool calling framework
   - Multi-agent coordination logic

2. **Scoped Tool Access**
   - Filter tools based on Steward recommendations
   - Dynamic tool loading for Butler context
   - Tool execution framework
   - Result aggregation

3. **Real-Time Reasoning Output**
   - Stream all Butler activities to reasoning output
   - Tool call progress indicators
   - Expert agent consultation messages
   - Wait time transparency

### Success Criteria
- [ ] Tatlock receives enriched requests (user + Steward notes)
- [ ] Only recommended tools are available
- [ ] Tatlock coordinates multiple tool calls
- [ ] All actions streamed to reasoning output
- [ ] Responses have consistent personality
- [ ] Synthesizes multi-source results coherently

### Estimated Effort
**4-5 weeks** - Complex coordination logic

---

## Phase 4: Expert Household Staff - Core Agents

**Goal**: Implement the initial set of domain-specific expert agents

### Priority Expert Agents

1. **The Librarian** (Research & Knowledge Management) ⭐ **Priority**
   - Research assistance and synthesis
   - Automatic research dossier generation
   - Knowledge base queries and organization
   - Reference management
   - Wiki integration (future: dedicated wiki container)
   - Mind map maintenance (future)
   - *Rationale: Helps guide development priorities through better research*

2. **The Developer** (Software Development)
   - Code generation assistance
   - Debugging support
   - Documentation generation
   - Architecture guidance
   - *Rationale: Directly supports building the system itself*

3. **The Handyman** (System Maintenance)
   - System status queries
   - Log analysis
   - Basic troubleshooting
   - Infrastructure monitoring

4. **The Secretary** (Scheduling & Organization)
   - Calendar integration (placeholder)
   - Task management (placeholder)
   - Reminder system
   - Schedule conflict detection

5. **The Housekeeper** (Home Automation)
   - Device control interface
   - Status queries
   - Automation triggers
   - Environmental monitoring

### Each Agent Includes
- Specialized prompt and personality
- Domain-specific tools
- MCP integration points (where applicable)
- Integration with Butler orchestration

### Success Criteria
- [ ] Each agent implemented as separate module
- [ ] Agents callable via tool framework
- [ ] Agents use specialized prompts
- [ ] Results integrate cleanly with Butler
- [ ] Can invoke specialized models (e.g., Codestral for Developer)

### Estimated Effort
**6-8 weeks** - Parallel development possible

---

## Phase 5: Persistence Layer - Database & Multi-Tenancy

**Goal**: Add persistent storage and multi-user support when needed

### Deliverables

1. **PostgreSQL Integration**
   - Docker compose configuration for PostgreSQL
   - Database schema design with tenant isolation
   - Alembic migrations setup
   - SQLAlchemy models

2. **Multi-Tenant Architecture**
   - Tenant identification middleware
   - Tenant-scoped database sessions
   - User authentication system (basic)
   - Per-tenant data isolation

3. **Core Data Models**
   - Users and tenants
   - Conversations and messages (migrate from in-memory)
   - Agent interactions log
   - System configuration and preferences

4. **Migration Strategy**
   - Gradual migration from in-memory to database
   - Backward compatibility during transition
   - Data export/import utilities

### Success Criteria
- [ ] PostgreSQL container running
- [ ] Multiple users can authenticate separately
- [ ] Each user sees only their own data
- [ ] Conversations persist across restarts
- [ ] Database migrations work correctly
- [ ] Tests verify tenant isolation

### Estimated Effort
**3-4 weeks** - Data layer foundation

### Why Later?
The core orchestration (Steward → Butler → Experts) can work entirely with in-memory state. We only need database persistence when we want conversations to survive restarts and multiple users to have isolated experiences.

---

## Phase 6: Extended Services Integration

**Goal**: Connect to additional supporting services

### Services to Integrate

1. **Redis (Memory & Caching)**
   - Docker compose setup
   - Conversation cache
   - Short-term memory
   - Session management

3. **Qdrant (Vector Storage)**
   - Docker compose setup
   - Long-term memory embeddings
   - Semantic search
   - Conversation history vectors

4. **SearxNG (Web Search)**
   - Docker compose setup
   - Search tool integration
   - Result processing
   - Privacy-preserving queries

### Success Criteria
- [ ] All services defined in docker-compose.yml
- [ ] Services communicate correctly
- [ ] Tatlock can invoke web search
- [ ] Redis used for session data
- [ ] Qdrant stores conversation embeddings
- [ ] Ollama serves the base model

### Estimated Effort
**3-4 weeks** - Infrastructure setup

---

## Phase 7: MCP (Model Context Protocol) Integration

**Goal**: Enable rich tool integrations via MCP

### Deliverables

1. **MCP Server Framework**
   - MCP server implementation
   - Tool registration via MCP
   - Schema validation
   - Error handling

2. **MCP Client in Agents**
   - PydanticAI MCP integration
   - Tool discovery from MCP servers
   - Dynamic tool loading
   - Result processing

3. **Initial MCP Tools**
   - File system operations
   - Database queries
   - API integrations
   - System commands

### Success Criteria
- [ ] MCP server running
- [ ] Tools exposed via MCP protocol
- [ ] Agents can discover and use MCP tools
- [ ] New tools addable without code changes
- [ ] MCP tools visible in Steward recommendations

### Estimated Effort
**3-4 weeks** - Standards-based integration

---

## Phase 8: Advanced Memory & Context

**Goal**: Implement sophisticated memory and context management

### Deliverables

1. **Long-Term Memory**
   - Conversation embedding pipeline
   - Semantic search over history
   - Memory consolidation
   - Relevance ranking

2. **Context Management**
   - Smart context window trimming
   - Conversation branching
   - Topic tracking
   - Memory retrieval integration

3. **Personalization**
   - User preference learning
   - Interaction pattern analysis
   - Adaptive responses
   - Custom agent personalities per user

### Success Criteria
- [ ] Conversations automatically embedded to Qdrant
- [ ] Relevant history retrieved for new requests
- [ ] Context stays within model limits
- [ ] User preferences affect responses
- [ ] Memory improves over time

### Estimated Effort
**4-5 weeks** - AI/ML heavy

---

## Phase 9: Extended Household Staff

**Goal**: Add specialized agents for additional domains

### Future Agents

1. **The Librarian** (Knowledge Management)
   - Personal documentation indexing
   - Research assistance
   - Knowledge base queries
   - Reference management

2. **The Accountant** (Financial Tracking)
   - Expense tracking
   - Budget monitoring
   - Financial reports
   - Transaction categorization

3. **The Chef** (Meal Planning)
   - Recipe management
   - Meal planning
   - Nutrition tracking
   - Grocery lists

4. **Others as Needed**
   - Domain-specific as requirements emerge

### Success Criteria
- [ ] Each new agent follows household pattern
- [ ] Integrates with Steward/Butler flow
- [ ] Has appropriate specialized tools
- [ ] Documented in PHILOSOPHY.md updates

### Estimated Effort
**Ongoing** - Add as needed

---

## Phase 10: User Experience Refinement

**Goal**: Polish the interaction experience

### Deliverables

1. **Personality Tuning**
   - Refine Tatlock's wit and tone
   - Consistent household character
   - Cultural references appropriate
   - Humor that doesn't annoy

2. **Transparency Improvements**
   - Better progress indicators
   - Clearer reasoning explanations
   - Informative wait messages
   - Error message clarity

3. **Performance Optimization**
   - Response time improvements
   - Model loading optimization
   - Caching strategies
   - Streaming smoothness

### Success Criteria
- [ ] Users find Tatlock engaging
- [ ] Wait times feel reasonable
- [ ] Errors are understandable
- [ ] System feels responsive

### Estimated Effort
**Ongoing** - Continuous improvement

---

## Phase 11: Production Hardening

**Goal**: Make the system production-ready for homelab deployment

### Deliverables

1. **Deployment**
   - Complete docker-compose stack
   - Environment configuration
   - Backup strategies
   - Update procedures

2. **Monitoring**
   - Health checks
   - Performance metrics
   - Error tracking
   - Usage analytics

3. **Security**
   - Authentication hardening
   - Rate limiting
   - Input validation
   - Audit logging

4. **Documentation**
   - Installation guide
   - Configuration reference
   - Troubleshooting guide
   - Architecture documentation

### Success Criteria
- [ ] One-command deployment
- [ ] System health is monitorable
- [ ] Secure for homelab use
- [ ] Well documented

### Estimated Effort
**3-4 weeks** - Production polish

---

## Dependencies Between Phases

```
Phase 1 (Ollama + PydanticAI) ← Foundation for all AI
    ↓
Phase 2 (Steward)
    ↓
Phase 3 (Butler/Tatlock)
    ↓
Phase 4 (Expert Agents) ← Phase 7 (MCP) can enhance
    ↓
Phase 5 (Database/Multi-Tenancy) ← Can be deferred
    ↓
Phase 6 (Extended Services) → Phase 8 (Advanced Memory)
    ↓
Phase 9 (Extended Staff) → Phase 10 (UX) → Phase 11 (Production)
```

**Critical Path**: Phases 1 → 2 → 3 → 4 must be sequential
**Can Be Deferred**: Phase 5 (Database) until you need persistence
**Parallel Opportunities**: Phase 6 and 7 can overlap; Phase 9 and 10 ongoing

---

## Overall Timeline Estimate

**Minimum Viable Household** (Phases 1-4): **15-20 weeks**
- Working Steward → Butler → Expert Agents with real LLM
- In-memory state (no persistence needed yet)
- Core household functional

**With Persistence** (Phases 1-5): **18-24 weeks**
- Add database and multi-tenancy
- Conversations survive restarts
- Multiple users supported

**Full-Featured System** (Phases 1-9): **35-45 weeks**
- All services integrated
- Advanced memory and context
- Extended household staff

**Production-Ready** (All phases): **40-50 weeks**
- Polished UX
- Hardened for homelab deployment
- Fully documented

*Note: Timeline assumes consistent part-time development effort*

---

## Success Metrics

### Technical
- System implements PHILOSOPHY.md patterns
- All household roles functional
- Multi-tenant isolation verified
- Real-time reasoning transparency working
- MCP integration complete

### User Experience
- Tatlock feels like interacting with a butler
- Wait times are transparent and acceptable
- Expert agents provide value in their domains
- System is reliable and trustworthy

### Architecture
- Clean separation between household roles
- Easy to add new agents/tools
- Model efficiency (base model stays loaded)
- Scales to household + friends usage

---

## Risk Management

### High Risk Items
1. **PydanticAI + Ollama integration complexity**
   - Mitigation: Prototype early, iterate on connection layer

2. **Multi-agent coordination complexity**
   - Mitigation: Start simple, add coordination gradually

3. **Model performance on homelab hardware**
   - Mitigation: Model selection, quantization, optimization

4. **Prompt engineering for personality consistency**
   - Mitigation: Extensive testing, user feedback, iteration

### Medium Risk Items
- MCP protocol adoption and tooling maturity
- Vector embedding quality for memory
- Home automation integration variability
- User authentication security

---

## Next Steps

1. **Immediate**: Commit model name fix (Tatlock)
2. **Week 1-2**: Begin Phase 1 (PostgreSQL + multi-tenancy design)
3. **Week 3**: Parallel prototype of Steward agent
4. **Ongoing**: Update this roadmap as we learn

---

**Document Status**: Active planning document
**Created**: 2025-12-06
**Last Updated**: 2025-12-06