# AI Orchestrator Implementation Plan

> **Project:** tower-of-joy AI Stack Enhancement
> **Created:** 2025-11-13
> **Status:** Phase 4 Complete ✅ - PydanticAI Migration Successful
> **Updated:** 2025-12-03
> **Framework:** PydanticAI with Ollama

## Executive Summary

This document outlines the implementation of a sophisticated AI orchestration layer using PydanticAI and aiohttp that provides an advanced agent system for the tower-of-joy homelab. The architecture provides:

**✅ MIGRATION COMPLETE (2025-11-26):**
Successfully migrated from LangChain/LangGraph to PydanticAI to achieve reliable tool calling with Ollama local models.

- **Advanced Memory Systems:** Three-tier memory with Qdrant for long-term semantic recall
- **Multi-Agent Workflows:** Intelligent routing to lightweight, heavy, and specialist models
- **Extensive Tool Integration:** Web search, file operations, calendar, home automation, image generation
- **Production-Ready API:** OpenAI-compatible endpoints for seamless Open WebUI integration
- **Superior Performance:** Proper context management, caching, and model selection

## Current vs Target Architecture

### Current Architecture (v0.6.0)

```
┌──────────────┐
│  Open WebUI  │
│  (Port 82)   │
└──────┬───────┘
       │
       │ Direct connection
       │
┌──────▼───────┐      ┌──────────────┐
│   Ollama     │      │   Qdrant     │
│  (Port 11434)│      │  (Port 6333) │
└──────────────┘      └──────────────┘
       │
       │ GPU inference
       │
┌──────▼───────┐
│  RTX 2080 Ti │
│   (11GB)     │
└──────────────┘
```

**Limitations:**
- Open WebUI's memory integration not working well
- No intelligent model routing
- Limited tool calling capabilities
- Single-model processing (no multi-agent coordination)
- Difficult to customize RAG behavior

### Target Architecture

```
┌────────────────────────────────────────────────────────────────┐
│                      User Interface Layer                       │
│                   Open WebUI (Port 82)                          │
└───────────────────────────┬────────────────────────────────────┘
                            │
                            │ /v1/chat/completions (OpenAI-compatible)
                            │
┌───────────────────────────▼────────────────────────────────────┐
│                  AI Orchestrator (Port 8084)                    │
│  ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓  │
│  ┃          aiohttp + PydanticAI Orchestration Layer       ┃  │
│  ┃  • OpenAI-compatible API wrapper                        ┃  │
│  ┃  • Request routing & agent coordination                 ┃  │
│  ┃  • Memory management (3-tier system)                    ┃  │
│  ┗━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛  │
│                           │                                     │
│    ┌──────────────────────┼──────────────────────┐             │
│    │                      │                      │             │
│  ┌─▼──────────┐   ┌───────▼───────┐   ┌─────────▼──────┐     │
│  │ Chat Agent │   │ Research Agent│   │  Tool Agent    │     │
│  │ (General)  │   │ (Deep search) │   │  (Actions)     │     │
│  └─────┬──────┘   └───────┬───────┘   └─────────┬──────┘     │
└────────┼──────────────────┼─────────────────────┼─────────────┘
         │                  │                     │
         │                  │                     │
┌────────▼──────────────────▼─────────────────────▼─────────────┐
│              Model Inference Layer (Ollama)                     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐        │
│  │ Lightweight  │  │    Heavy     │  │  Specialist  │        │
│  │ gemma:2b     │  │ mistral:7b   │  │ codestral    │        │
│  │ gemma:7b     │  │ gemma2:9b    │  │ codegemma    │        │
│  │              │  │ gemma2:27b   │  │ mixtral:8x7b │        │
│  │ ~2-4GB VRAM  │  │ ~6-8GB VRAM  │  │ ~8-10GB VRAM │        │
│  └──────────────┘  └──────────────┘  └──────────────┘        │
└───────────────────────────────────────────────────────────────┘
         │
         │ GPU acceleration
         │
┌────────▼───────────────────────────────────────────────────────┐
│                     NVIDIA RTX 2080 Ti (11GB VRAM)              │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────┐    ┌───────────────────────────────┐
│   Memory & Storage      │    │   External Tools & APIs       │
│   (Qdrant Port 6333)    │    │                               │
│                         │    │  • Core API (web scraper)     │
│  Tier 1: Working Memory │    │  • Nextcloud API (files)      │
│  Tier 2: Summaries      │    │  • Calendar (CalDAV)          │
│  Tier 3: Vector Store   │    │  • ComfyUI (image gen)        │
│  • Conversations        │    │  • Home Assistant (IoT)       │
│  • Documents            │    │  • Web Search (DuckDuckGo)    │
│  • User facts           │    │  • Future integrations        │
└─────────────────────────┘    └───────────────────────────────┘
```

**Advantages:**
- ✅ Intelligent multi-model routing (right model for the task)
- ✅ Proper conversation memory with semantic recall
- ✅ Multi-agent coordination for complex tasks
- ✅ Extensive tool calling (web search, files, calendar, automation)
- ✅ Research agents for deep information gathering
- ✅ Image generation via ComfyUI integration
- ✅ Gradual migration path (run parallel with current setup)
- ✅ Foundation for custom mobile apps later

## Technology Stack

### Core Framework (Current - Post-Migration)
- **PydanticAI** - Modern AI agent framework with type safety
- **Ollama Native SDK ≥0.4.0** - Direct Ollama integration with tool calling
- **aiohttp 3.10.1** - Async HTTP server framework
- **aiohttp-cors 0.7.0** - CORS support for web clients
- **Pydantic ≥2.10.3** - Request/response validation and type safety
- **pydantic-settings 2.6.1** - Configuration management

### AI & Memory
- **pydantic-ai-slim** - Minimal PydanticAI library
- **qdrant-client ≥1.12.0** - Vector database client
- **Ollama** - Local LLM inference (native integration)

### Utilities
- **httpx ≥0.28.0** - Async HTTP client for external APIs
- **python-dotenv ≥1.1.0** - Environment configuration
- **pytz ≥2025.2** - Timezone support for date/time tools

### Container
- **Python 3.12** - Runtime
- **Docker** - Containerization
- **Network:** ai-dataplane (shared with Ollama, Qdrant, Open WebUI)

## Three-Tier Memory Architecture

### Tier 1: Working Memory (In-Memory)
**Purpose:** Immediate context for ongoing conversation

**Implementation:** `ConversationBufferMemory`
- Stores last 10 conversation turns in RAM
- Fast access (< 1ms)
- Automatic pruning when limit reached
- Lost on container restart (ephemeral)

**Storage:** 0MB persistent, ~5KB RAM

### Tier 2: Short-Term Memory (SQLite)
**Purpose:** Recent conversation summaries

**Implementation:** `ConversationSummaryMemory`
- Summarized conversation history (hours to days)
- Stored in SQLite database
- Medium access speed (~10ms)
- Persists across restarts

**Storage:** `~/docker-data/ai-orchestrator/data/memory.db` (~500KB per 100 conversations)

### Tier 3: Long-Term Memory (Qdrant)
**Purpose:** Semantic search across entire conversation history

**Implementation:** `VectorStoreRetrieverMemory` with Qdrant
- All conversations embedded and stored as vectors
- Semantic similarity search for relevant context
- Unlimited history retention
- Fast semantic search (< 50ms)

**Storage:** Qdrant collection `conversation_memory` (~1KB per turn, 10MB for 10k turns)

### Memory Consolidation Strategy

```python
# Consolidation triggers
CONSOLIDATION_RULES = {
    "message_count": 10,        # Every 10 messages → Summarize to Tier 2
    "token_limit": 2000,        # When context > 2000 tokens → Compress
    "conversation_end": True,   # End of conversation → Embed to Tier 3
    "explicit_save": True,      # User: "remember this" → Force save
}
```

## Multi-Agent Workflow System

### Router Agent (Lightweight Model)
**Model:** gemma:2b or gemma:7b
**Purpose:** Analyze incoming requests and route to appropriate agent/model

**Decision Criteria:**
- Task complexity (token estimation, keyword analysis)
- Domain specialization (code, math, general, creative)
- Tool requirements (web search, file access, image generation)
- Response quality needs (fast vs accurate)

### Chat Agent (General Purpose)
**Model:** mistral:7b (default) or gemma:7b (simple queries)
**Purpose:** Handle general conversation, Q&A, casual interactions

**Capabilities:**
- Normal chat interactions
- Simple questions and answers
- Memory recall from Qdrant
- Basic tool calling (web search, file access)

### Research Agent (Deep Analysis)
**Model:** mixtral:8x7b or mistral:7b
**Purpose:** Complex research tasks requiring web search and synthesis

**Workflow:**
1. Query expansion (generate related search terms)
2. Web search (DuckDuckGo, multiple queries)
3. Content scraping (via Core API)
4. Analysis (extract key information)
5. Synthesis (generate comprehensive report)

**Tools:**
- Web search
- Web scraping (Core API)
- Document retrieval (Qdrant)

### Code Agent (Specialist)
**Model:** codestral:latest or codegemma:latest
**Purpose:** Programming tasks, debugging, code generation

**Capabilities:**
- Code generation (multiple languages)
- Debugging and optimization
- Code explanation
- API integration examples

### Tool Agent (Action Executor)
**Model:** mistral:7b
**Purpose:** Execute actions using external tools and APIs

**Available Tools:**
- **Web Search** (DuckDuckGo)
- **Web Scraping** (Core API)
- **File Operations** (Nextcloud API)
- **Calendar Management** (CalDAV via Nextcloud)
- **Image Generation** (ComfyUI/Stable Diffusion)
- **Home Automation** (Home Assistant - future)
- **Task Management** (future custom system)

## OpenAI-Compatible API Design

### Endpoint: POST /v1/chat/completions

**Request Schema:**
```json
{
  "model": "gpt-3.5-turbo",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Hello!"}
  ],
  "stream": false,
  "temperature": 0.7,
  "max_tokens": 2048
}
```

**Model Aliasing:**
```python
MODEL_ALIASES = {
    "gpt-3.5-turbo": "gemma:7b",           # Fast, lightweight
    "gpt-4": "mistral:7b",                 # High quality
    "gpt-4-turbo": "mixtral:8x7b",         # Very capable
    "gpt-4-code": "codestral:latest",      # Code specialist
    "gpt-4-32k": "gemma2:27b",             # Longer context
}
```

**Response Schema (Non-Streaming):**
```json
{
  "id": "chatcmpl-1234567890",
  "object": "chat.completion",
  "created": 1699564800,
  "model": "gpt-3.5-turbo",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Hello! How can I help you?"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 8,
    "total_tokens": 18
  }
}
```

**Response Schema (Streaming):**
```
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
```

### Additional Endpoints

- `GET /v1/models` - List available models
- `GET /health` - Health check
- `GET /metrics` - Prometheus metrics
- `POST /v1/embeddings` - Generate embeddings (future)

## Tool Integration Plan

### Phase 1 Tools (Core Functionality)

#### 1. Web Search Tool
**Integration:** DuckDuckGo API (no API key required)
**Purpose:** Find current information on the web

```python
async def web_search(query: str, num_results: int = 5) -> str:
    """Search the web for current information."""
    # Implementation using duckduckgo-search
```

#### 2. Web Scraping Tool
**Integration:** Core API (already deployed)
**Purpose:** Extract content from web pages

```python
async def scrape_webpage(url: str) -> str:
    """Extract clean text content from a URL."""
    response = await httpx.post(
        "http://core-api:8083/scrape",
        json={"url": url}
    )
```

#### 3. Document Search Tool
**Integration:** Qdrant documents collection
**Purpose:** Search uploaded documents and previous conversations

```python
async def search_documents(query: str) -> str:
    """Search through uploaded documents and conversation history."""
    # Semantic search in Qdrant
```

### Phase 2 Tools (Productivity)

#### 4. Nextcloud File Tool
**Integration:** Nextcloud WebDAV API
**Purpose:** Search and access files in Nextcloud

```python
async def search_files(query: str) -> str:
    """Search for files in Nextcloud."""
```

#### 5. Calendar Tool
**Integration:** CalDAV via Nextcloud
**Purpose:** Check calendar, add events

```python
async def check_calendar(date_range: str = "today") -> str:
    """Check calendar for events or add new events."""
```

### Phase 3 Tools (Advanced)

#### 6. Image Generation Tool
**Integration:** ComfyUI API (when deployed)
**Purpose:** Generate images from text descriptions

```python
async def generate_image(description: str) -> str:
    """Generate images using Stable Diffusion."""
```

#### 7. Home Automation Tool
**Integration:** Home Assistant API (when deployed)
**Purpose:** Control smart home devices

```python
async def control_home(device: str, action: str) -> str:
    """Control smart home devices."""
```

#### 8. Task Management Tool
**Integration:** Custom task system (future)
**Purpose:** Create, read, update tasks and reminders

```python
async def manage_tasks(action: str, task: str = None) -> str:
    """Create and manage tasks and reminders."""
```

## Directory Structure

```
/home/jpmschweitzer/Projects/portainer-core/
├── services/
│   └── core-ai/
│       ├── Dockerfile
│       ├── requirements.txt
│       ├── .env.example
│       ├── README.md
│       ├── main.py              # aiohttp app entry point
│       └── src/
│           ├── __init__.py
│           ├── config.py        # Configuration management
│           ├── prompts.py       # System prompt variants
│           ├── utils.py         # Common utilities
│           │
│           ├── agents/          # Agent implementations
│           │   ├── __init__.py
│           │   ├── pydantic_agent.py  # Main PydanticAI agent
│           │   ├── simple.py          # Fallback LiteLLM agent
│           │   └── ollama_native_agent.py  # Native Ollama agent
│           │
│           ├── memory/          # Memory systems
│           │   ├── __init__.py
│           │   ├── manager.py   # Memory manager
│           │   ├── types.py     # Memory types and enums
│           │   └── consolidation.py  # Memory consolidation
│           │
│           ├── tools/           # Tool implementations
│           │   ├── __init__.py
│           │   ├── local.py     # Local tools (time, services)
│           │   ├── registry.py  # Tool registry
│           │   └── openapi_discovery.py  # OpenAPI tool discovery
│           │
│           └── tests/           # Test suite
│               ├── __init__.py
│               ├── test_ai_flow_quality.py
│               └── reports/
│
└── stacks/
    └── core-ai.yml              # Docker Compose stack

/home/jpmschweitzer/docker-data/
└── core-ai/
    ├── data/
    │   ├── memory.db            # SQLite for summaries
    │   └── checkpoints/         # Agent checkpoints
    ├── logs/
    │   └── app.log              # Application logs
    └── cache/                   # Response cache
```

## Implementation Phases

### Phase 1: Foundation (Week 1) ✅ **COMPLETED 2025-11-13**
**Goal:** Basic OpenAI-compatible API wrapper that works with Open WebUI

**Status:** ✅ All tasks completed. Implementation added to Core API service.

**Tasks:** ✅ **ALL COMPLETE**
1. ✅ Create service directory structure (extended Core API instead)
2. ✅ Implement aiohttp app with `/v1/chat/completions` endpoint
3. ✅ Add OpenAI request/response schemas (Pydantic models)
4. ✅ Connect to Ollama for model inference
5. ✅ Implement basic streaming support (SSE format)
6. ✅ Add model aliasing (gpt-3.5-turbo → gemma:7b)
7. ✅ Create Dockerfile and requirements.txt
8. ✅ Create Docker Compose stack definition
9. ✅ Deploy to ai-dataplane network
10. ✅ Test with Open WebUI

**Deliverables:** ✅ **ALL DELIVERED**
- ✅ Working `/v1/chat/completions` endpoint
- ✅ Both streaming and non-streaming responses
- ✅ Model name mapping
- ✅ Docker container deployed (core-ai on port 8084)
- ✅ `/v1/models` endpoint
- ✅ Ollama client with connection management

**Success Criteria:** ✅ **ALL MET**
- ✅ OpenAI-compatible API responding correctly
- ✅ Streaming works properly (Server-Sent Events format)
- ✅ Non-streaming responses working
- ✅ Model aliasing functional
- ✅ Health check passing, Ollama connectivity verified

### Phase 2: Memory Systems (Week 2) ✅ **COMPLETED 2025-11-23**
**Goal:** Persistent conversation memory with three-tier architecture

**Status:** ✅ Complete. Three-tier memory system implemented.

**Completed Tasks:**
1. ✅ Implement Tier 1: ConversationBufferMemory (in-memory)
2. ✅ Implement Tier 2: ConversationSummaryMemory (SQLite)
3. ✅ Integrate Tier 3: VectorStoreRetrieverMemory (Qdrant)
4. ✅ Create Qdrant collections (conversation_memory, documents, user_facts)
5. ✅ Implement memory consolidation service
6. ✅ Add conversation history API endpoints
7. ✅ Build memory recall in conversation flow
8. ✅ Test memory persistence across container restarts
9. ✅ Add memory metrics

**Deliverables:** ✅ **ALL DELIVERED**
- ✅ Three-tier memory system
- ✅ Persistent conversation storage
- ✅ Memory consolidation pipeline
- ✅ Conversation recall functionality
- ✅ Memory metrics

**Success Criteria:** ✅ **ALL MET**
- ✅ Conversations persist across restarts
- ✅ Memory recall provides relevant context
- ✅ Semantic search returns appropriate results
- ✅ No memory leaks or unbounded growth

### Phase 3: Research Capabilities ✅ **COMPLETE** (2025-11-24)
**Goal:** Web search and research workflows

**Status:** ✅ **COMPLETE** - All success criteria met

**Completed Tasks:**
1. ✅ Integrated DuckDuckGo web search API
2. ✅ Implemented automatic content scraping from search results
3. ✅ Added web_search tool (search + scrape in one call)
4. ✅ Added web_scrape tool (targeted URL extraction)
5. ✅ Enhanced system prompts for research query detection
6. ✅ Added enhanced progress indicators
7. ✅ Comprehensive testing (100% success rate)
8. ✅ Tool invocation logging for debugging
9. ✅ System prompt optimization
10. ✅ Model validation (mistral:7b confirmed best for tools)

**Deliverables:** ✅ **ALL DELIVERED**
- ✅ Web search with DuckDuckGo integration
- ✅ Automatic content extraction from results
- ✅ Research query detection in system prompts
- ✅ Multiple tools (infrastructure + research)
- ✅ Enhanced user experience with visual feedback
- ✅ Comprehensive test suite

**Success Criteria:** ✅ **ALL MET**
- ✅ Research detection accuracy: 100%
- ✅ Average response time: 5.3s (target: <10s)
- ✅ Source citation rate: 100%
- ✅ Tool calling reliability: 100% on complex queries
- ✅ No regression in existing functionality

### Phase 4: Framework Migration ✅ **COMPLETE** (2025-11-26)
**Goal:** Migrate from LangChain/LangGraph to PydanticAI for improved reliability

**Status:** ✅ **COMPLETE** - All success criteria met

**Completed Tasks:**
1. ✅ Updated requirements.txt (removed langchain*, added pydantic-ai)
2. ✅ Rewrote agent to use PydanticAI with native Ollama SDK
3. ✅ Converted all tools to PydanticAI format
4. ✅ Updated streaming for PydanticAI event format
5. ✅ Optimized model selection (mistral-tools:7b)
6. ✅ Created PydanticAI-optimized system prompts
7. ✅ Enhanced health checks for agent monitoring
8. ✅ Production testing and validation
9. ✅ Documentation updates

**Deliverables:** ✅ **ALL DELIVERED**
- ✅ PydanticAI integration with native Ollama SDK
- ✅ All tools migrated to PydanticAI format
- ✅ Model optimized for tool calling
- ✅ PydanticAI-optimized system prompts
- ✅ Improved streaming consistency
- ✅ Enhanced agent health monitoring
- ✅ Complete documentation

**Success Criteria:** ✅ **ALL MET**
- ✅ Tool calling works reliably
- ✅ All models supported (Gemma, Mistral, etc.)
- ✅ Streaming output consistent and clean
- ✅ No regressions in memory system
- ✅ API endpoints unchanged (backward compatible)
- ✅ Performance within targets

**Implementation Details:**
- **Framework:** PydanticAI with native Ollama SDK
- **Model:** mistral-tools:7b (optimized for tool calling)
- **Prompt:** Optimized variants for PydanticAI
- **Tools:** All tools as PydanticAI-compatible functions
- **Migration Time:** ~6 hours (as estimated)

**Performance (Post-Migration):**
- Simple queries: ~0.3-1s
- Tool-using queries: ~2-5s (improved from LangChain)
- Research queries: ~4-7s (maintained)
- VRAM usage: ~4-6GB with mistral-tools:7b

**Why This Migration:**
- ❌ **Problem:** LangChain/LangGraph failed to trigger tools reliably with Ollama
- ❌ **Problem:** Gemma models returned errors with LangChain
- ❌ **Problem:** Inconsistent streaming behavior
- ✅ **Solution:** PydanticAI has native Ollama support with excellent type safety
- ✅ **Result:** Reliable tool calling across all models

### Phase 5: Multi-Agent Patterns & RAG Optimization (Future)
**Goal:** Enhanced multi-agent workflows and advanced RAG capabilities

**Tasks:**
1. Implement router agent pattern for intelligent task routing
2. Create specialized agents (code, research, tool execution)
3. Build agent coordination system
4. Implement hybrid retrieval (dense + sparse)
5. Create document embedding pipeline
6. Build RAG chain with Qdrant
7. Add re-ranking for better results
8. Integrate RAG with conversation flow
9. Add document upload endpoint
10. Optimize retrieval performance

**Deliverables:**
- Multi-agent routing system
- Specialized agent implementations
- Hybrid search system (semantic + keyword)
- Document embedding pipeline
- RAG-enhanced responses
- Re-ranking optimization
- Document upload API

**Success Criteria:**
- Router correctly selects appropriate agent
- Specialized agents outperform general agent
- Documents can be uploaded and indexed
- Semantic search returns relevant chunks
- Hybrid search improves accuracy
- RAG responses use document context
- Performance meets targets (< 100ms retrieval)

### Phase 6: Production Hardening (Future)
**Goal:** Observability, error handling, optimization

**Tasks:**
1. Add structured logging (structlog)
2. Implement comprehensive error handling
3. Add retry logic for external calls
4. Implement Prometheus metrics
5. Create health check endpoints
6. Add request/response caching
7. Optimize model selection logic
8. Performance testing and optimization
9. Load testing (concurrent requests)
10. Documentation and deployment guide

**Deliverables:**
- Production-ready service
- Monitoring and metrics
- Error handling
- Performance benchmarks
- Load test results
- Complete documentation

**Success Criteria:**
- Structured logs for debugging
- All errors handled gracefully
- Metrics exported to Prometheus
- Health checks pass
- Response times < 2s (p95)
- Can handle 10+ concurrent requests
- Documentation complete

## Docker Configuration

### Dockerfile

```dockerfile
FROM python:3.12-slim

# Prevent Python from writing pyc files and buffering
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONPATH=/app

WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements first (better caching)
COPY requirements.txt .
RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY ./src /app/src
COPY ./main.py /app/

# Create non-root user
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /app
USER appuser

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
    CMD curl -f http://localhost:8084/health || exit 1

# Expose port
EXPOSE 8084

# Run aiohttp server
CMD ["python", "main.py"]
```

### Docker Compose Stack (stacks/core-ai.yml)

```yaml
version: '3.8'

# Core AI - PydanticAI service with OpenAI-compatible API
# Purpose: Intelligent agent workflows with multi-model routing and advanced memory
# Port: 8084 (HTTP API)
# Network: ai-dataplane (shared with Ollama, Qdrant, Open WebUI)
# Dependencies: Ollama (models), Qdrant (memory)

services:
  core-ai:
    build:
      context: /home/jpmschweitzer/Projects/portainer-core/services/core-ai
      dockerfile: Dockerfile
    container_name: core-ai
    restart: unless-stopped

    ports:
      - "8084:8084"

    environment:
      # Application
      - APP_NAME=Core AI
      - APP_VERSION=1.0.0
      - DEBUG=false
      - LOG_LEVEL=INFO
      - ENVIRONMENT=production

      # Server
      - HOST=0.0.0.0
      - PORT=8084

      # Model endpoints
      - OLLAMA_BASE_URL=http://ollama:11434
      - QDRANT_URL=http://qdrant:6333

      # Model configuration
      - DEFAULT_MODEL=mistral-tools:7b
      - AGENT_MODEL=mistral-tools:7b

      # Memory configuration
      - MEMORY_ENABLED=true
      - MEMORY_TIER1_SIZE=10
      - DEFAULT_USER_ID=default

      # Agent configuration
      - ENABLE_TOOLS=true
      - SYSTEM_PROMPT_VARIANT=minimal_agent

    volumes:
      # Persistent data (SQLite, checkpoints)
      - /home/jpmschweitzer/docker-data/core-ai/data:/app/data

      # Logs
      - /home/jpmschweitzer/docker-data/core-ai/logs:/app/logs

      # Cache
      - /home/jpmschweitzer/docker-data/core-ai/cache:/app/cache

    networks:
      - ai-dataplane

    depends_on:
      - ollama
      - qdrant

    labels:
      - "com.centurylinklabs.watchtower.enable=true"

    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8084/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

networks:
  ai-dataplane:
    external: true
```

## Performance Targets

### Response Time Targets

| Scenario | Target (p50) | Target (p95) | Target (p99) |
|----------|--------------|--------------|--------------|
| Simple chat (lightweight model) | < 500ms | < 1s | < 2s |
| Complex chat (heavy model) | < 1s | < 2s | < 4s |
| Research task (multi-step) | < 5s | < 10s | < 15s |
| Tool calling (web search) | < 2s | < 4s | < 6s |
| RAG retrieval | < 100ms | < 200ms | < 500ms |

### Throughput Targets

| Metric | Target |
|--------|--------|
| Concurrent requests | 10+ |
| Requests per minute | 60+ |
| GPU utilization | 60-80% |
| Memory usage (orchestrator) | < 1GB |

### Quality Targets

| Metric | Target |
|--------|--------|
| Model routing accuracy | > 90% |
| Memory recall relevance | > 85% |
| Tool calling success rate | > 95% |
| API compatibility | 100% (OpenAI spec) |

## Monitoring & Observability

### Metrics to Track (Prometheus)

**Request Metrics:**
- `core_ai_requests_total{model, status}` - Total requests
- `core_ai_request_duration_seconds{model}` - Request latency histogram
- `core_ai_active_requests` - Currently processing requests

**Agent Metrics:**
- `core_ai_agent_invocations_total{agent_type}` - Agent usage
- `core_ai_agent_duration_seconds{agent_type}` - Agent execution time
- `core_ai_model_routing_total{from_model, to_model}` - Routing decisions

**Memory Metrics:**
- `core_ai_memory_consolidations_total` - Memory consolidations
- `core_ai_memory_retrieval_duration_seconds` - Retrieval time
- `core_ai_memory_size_bytes{tier}` - Memory size per tier

**Tool Metrics:**
- `core_ai_tool_calls_total{tool_name, status}` - Tool usage
- `core_ai_tool_duration_seconds{tool_name}` - Tool execution time
- `core_ai_tool_errors_total{tool_name}` - Tool failures

### Logging Strategy

**Structured Logging:**
```python
logger.info(
    "chat_request_received",
    request_id=request_id,
    model=request.model,
    message_count=len(request.messages),
    stream=request.stream,
    user_id=user_id
)
```

**Log Levels:**
- **DEBUG:** Detailed agent workflows, tool calls, memory operations
- **INFO:** Request/response, agent routing decisions
- **WARNING:** Fallbacks, retries, degraded performance
- **ERROR:** Failures, exceptions, unrecoverable errors

### Health Checks

**Endpoint:** `GET /health`

**Checks:**
- API status
- Ollama connectivity
- Qdrant connectivity
- Memory system health
- Disk space

**Response:**
```json
{
  "status": "healthy",
  "timestamp": 1699564800,
  "checks": {
    "api": "healthy",
    "ollama": "healthy",
    "qdrant": "healthy",
    "memory": "healthy",
    "disk": "healthy"
  },
  "version": "1.0.0"
}
```

## Security Considerations

### Authentication (Phase 6)

Optional API key authentication:
```python
async def verify_api_key(request):
    api_key = request.headers.get("Authorization")
    if not api_key:
        raise web.HTTPUnauthorized()
    # Verify API key
```

### Rate Limiting (Phase 6)

Prevent abuse:
```python
# Implement rate limiting middleware
# 60 requests per minute per IP
```

### Input Validation

Pydantic models validate all inputs:
```python
class ChatCompletionRequest(BaseModel):
    model: str = Field(min_length=1, max_length=100)
    messages: List[ChatMessage]
    max_tokens: Optional[int] = Field(ge=1, le=4096, default=None)
```

### CORS Configuration

Restrict origins:
```python
cors = aiohttp_cors.setup(app, defaults={
    "http://192.168.86.149:82": aiohttp_cors.ResourceOptions(
        allow_credentials=True,
        expose_headers="*",
        allow_headers="*",
    )
})
```

## Testing Strategy

### Unit Tests
- Test individual components (agents, memory, tools)
- Mock external dependencies (Ollama, Qdrant)
- Use pytest and pytest-asyncio

### Integration Tests
- Test complete API endpoints
- Real connections to Ollama/Qdrant
- Test streaming and non-streaming responses

### Load Tests
- Test concurrent request handling
- Measure response times under load
- Identify bottlenecks

### End-to-End Tests
- Test via Open WebUI
- Test complex multi-agent workflows
- Test tool calling and RAG

## Success Criteria

### Phase 1 Success ✅
- ✅ Open WebUI can connect and chat
- ✅ Streaming works correctly
- ✅ Model aliases function
- ✅ No errors in logs

### Phase 2 Success ✅
- ✅ Memory persists across restarts
- ✅ Semantic recall works
- ✅ Consolidation triggers properly
- ✅ No memory leaks

### Phase 3 Success ✅
- ✅ Web search functional
- ✅ Content extraction works
- ✅ Research query detection accurate
- ✅ Tool calling reliable

### Phase 4 Success ✅
- ✅ PydanticAI integration complete
- ✅ Tools callable from agents
- ✅ Streaming works properly
- ✅ No regressions

### Phase 5 Success (Future)
- Multi-agent routing functional
- Specialized agents outperform general agent
- Documents indexed and searchable
- Hybrid search improves results
- RAG provides relevant context
- Performance targets met

### Phase 6 Success (Future)
- Metrics exported to Prometheus
- Health checks pass
- Load tests successful
- Documentation complete
- Production deployment successful

## Future Enhancements (Post-Launch)

### Phase 7+: Advanced Features
1. **Nextcloud Integration** - File search, calendar management
2. **ComfyUI Integration** - Image generation capabilities
3. **Home Assistant Integration** - Smart home control
4. **Task Management System** - Custom task/todo system
5. **Mobile Apps** - Native iOS/Android apps
6. **Home Screen Widgets** - Quick actions and status
7. **Voice Interface** - Voice command processing
8. **Proactive Notifications** - Intelligent reminders
9. **Multi-User Support** - Per-user memory and preferences
10. **Fine-Tuned Models** - Custom models for specific tasks

## Risk Assessment & Mitigation

### Risk 1: GPU VRAM Exhaustion
**Impact:** High - Service fails if VRAM exceeded
**Probability:** Medium - Can happen with concurrent heavy model loads
**Mitigation:**
- Implement model queue (max 1-2 concurrent)
- Use quantized models (Q4, Q5)
- Monitor VRAM usage
- Automatic fallback to CPU for lightweight models

### Risk 2: Qdrant Performance Degradation
**Impact:** Medium - Slower retrieval affects UX
**Probability:** Low - Qdrant is fast with proper indexing
**Mitigation:**
- Use HNSW indexing (default)
- Implement collection partitioning
- Add query filters to reduce search space
- Cache frequent queries

### Risk 3: OpenAI API Incompatibility
**Impact:** High - Open WebUI won't work
**Probability:** Low - Spec is well-defined
**Mitigation:**
- Follow OpenAI API spec exactly
- Test thoroughly with Open WebUI
- Document unsupported features
- Keep Ollama direct as fallback

### Risk 4: Complex Agent Workflows Timeout
**Impact:** Medium - Some tasks fail
**Probability:** Medium - Research tasks can be slow
**Mitigation:**
- Set reasonable timeouts (5 minutes)
- Implement streaming progress updates
- Break down complex tasks
- Return partial results on timeout

### Risk 5: Memory Consolidation Overhead
**Impact:** Low - Slight performance impact
**Probability:** High - Consolidation is CPU intensive
**Mitigation:**
- Run consolidation async (background)
- Batch consolidation operations
- Use lightweight model for summaries
- Monitor consolidation performance

## Conclusion

This implementation plan provides a comprehensive roadmap for building a sophisticated AI orchestration layer that transforms the tower-of-joy infrastructure from a basic LLM chat interface into an intelligent, multi-agent system with proper memory, tool integration, and research capabilities.

The phased approach ensures steady progress with testable milestones. The architecture is designed to integrate seamlessly with existing infrastructure (Ollama, Qdrant) while providing a foundation for future enhancements like mobile apps and home automation.

**Current Status (Phase 4 Complete):**
- ✅ OpenAI-compatible API for Open WebUI
- ✅ Three-tier memory system with semantic recall
- ✅ Web search and research capabilities
- ✅ PydanticAI framework with reliable tool calling
- ✅ Production-grade performance and reliability

**Next Steps:**
- Phase 5: Multi-agent patterns and RAG optimization
- Phase 6: Production hardening with monitoring
- Phase 7+: Advanced integrations (Nextcloud, ComfyUI, Home Assistant)

This establishes the tower-of-joy project as a cutting-edge AI homelab with capabilities rivaling commercial solutions, all running on local hardware with full data sovereignty.

---

**Plan Status:** ✅ Phase 4 Complete - PydanticAI Migration Successful
**Completed Phases:**
- Phase 1: Foundation (2025-11-13)
- Phase 2: Memory Systems (2025-11-23)
- Phase 3: Research Capabilities (2025-11-24)
- Phase 4: Framework Migration to PydanticAI (2025-11-26)

**Next Steps:** Phase 5 - Multi-Agent Patterns & RAG Optimization (Future)