# AI Orchestrator Implementation Plan

> **Project:** tower-of-joy AI Stack Enhancement
> **Created:** 2025-11-13
> **Status:** Phase 1 Complete ✅ - Phase 2 In Progress 🔄
> **Updated:** 2025-11-13
> **Target Completion:** 5 weeks remaining (Phase 2-6)

## Executive Summary

This document outlines the plan to build a sophisticated AI orchestration layer using LangGraph and FastAPI that will replace Open WebUI's direct connection to Ollama. The new architecture provides:

- **Advanced Memory Systems:** Three-tier memory with Qdrant for long-term semantic recall
- **Multi-Agent Workflows:** Intelligent routing to lightweight, heavy, and specialist models
- **Extensive Tool Integration:** Web search, file operations, calendar, home automation, image generation
- **Production-Ready API:** OpenAI-compatible endpoints for seamless Open WebUI integration
- **Superior Performance:** Proper context management, caching, and model selection

## Current vs Target Architecture

### Current Architecture (v0.6.0)

```
┌──────────────┐
│  Open WebUI  │
│  (Port 82)   │
└──────┬───────┘
       │
       │ Direct connection
       │
┌──────▼───────┐      ┌──────────────┐
│   Ollama     │      │   Qdrant     │
│  (Port 11434)│      │  (Port 6333) │
└──────────────┘      └──────────────┘
       │
       │ GPU inference
       │
┌──────▼───────┐
│  RTX 2080 Ti │
│   (11GB)     │
└──────────────┘
```

**Limitations:**
- Open WebUI's memory integration not working well
- No intelligent model routing
- Limited tool calling capabilities
- Single-model processing (no multi-agent coordination)
- Difficult to customize RAG behavior

### Target Architecture

```
┌────────────────────────────────────────────────────────────────┐
│                      User Interface Layer                       │
│                   Open WebUI (Port 82)                          │
└───────────────────────────┬────────────────────────────────────┘
                            │
                            │ /v1/chat/completions (OpenAI-compatible)
                            │
┌───────────────────────────▼────────────────────────────────────┐
│                  AI Orchestrator (Port 8084)                    │
│  ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓  │
│  ┃          FastAPI + LangGraph Orchestration Layer        ┃  │
│  ┃  • OpenAI-compatible API wrapper                        ┃  │
│  ┃  • Request routing & agent coordination                 ┃  │
│  ┃  • Memory management (3-tier system)                    ┃  │
│  ┗━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛  │
│                           │                                     │
│    ┌──────────────────────┼──────────────────────┐             │
│    │                      │                      │             │
│  ┌─▼──────────┐   ┌───────▼───────┐   ┌─────────▼──────┐     │
│  │ Chat Agent │   │ Research Agent│   │  Tool Agent    │     │
│  │ (General)  │   │ (Deep search) │   │  (Actions)     │     │
│  └─────┬──────┘   └───────┬───────┘   └─────────┬──────┘     │
└────────┼──────────────────┼─────────────────────┼─────────────┘
         │                  │                     │
         │                  │                     │
┌────────▼──────────────────▼─────────────────────▼─────────────┐
│              Model Inference Layer (Ollama)                     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐        │
│  │ Lightweight  │  │    Heavy     │  │  Specialist  │        │
│  │ gemma:2b     │  │ mistral:7b   │  │ codestral    │        │
│  │ gemma:7b     │  │ gemma2:9b    │  │ codegemma    │        │
│  │              │  │ gemma2:27b   │  │ mixtral:8x7b │        │
│  │ ~2-4GB VRAM  │  │ ~6-8GB VRAM  │  │ ~8-10GB VRAM │        │
│  └──────────────┘  └──────────────┘  └──────────────┘        │
└───────────────────────────────────────────────────────────────┘
         │
         │ GPU acceleration
         │
┌────────▼───────────────────────────────────────────────────────┐
│                     NVIDIA RTX 2080 Ti (11GB VRAM)              │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────┐    ┌───────────────────────────────┐
│   Memory & Storage      │    │   External Tools & APIs       │
│   (Qdrant Port 6333)    │    │                               │
│                         │    │  • Core API (web scraper)     │
│  Tier 1: Working Memory │    │  • Nextcloud API (files)      │
│  Tier 2: Summaries      │    │  • Calendar (CalDAV)          │
│  Tier 3: Vector Store   │    │  • ComfyUI (image gen)        │
│  • Conversations        │    │  • Home Assistant (IoT)       │
│  • Documents            │    │  • Web Search (DuckDuckGo)    │
│  • User facts           │    │  • Future integrations        │
└─────────────────────────┘    └───────────────────────────────┘
```

**Advantages:**
- ✅ Intelligent multi-model routing (right model for the task)
- ✅ Proper conversation memory with semantic recall
- ✅ Multi-agent coordination for complex tasks
- ✅ Extensive tool calling (web search, files, calendar, automation)
- ✅ Research agents for deep information gathering
- ✅ Image generation via ComfyUI integration
- ✅ Gradual migration path (run parallel with current setup)
- ✅ Foundation for custom mobile apps later

## Technology Stack

### Core Framework
- **LangGraph 0.2.60** - Stateful multi-agent orchestration (not basic LangChain)
- **FastAPI 0.115.0** - REST API framework
- **Uvicorn 0.32.0** - ASGI server
- **Pydantic 2.10.4** - Request/response validation

### AI & Memory
- **langchain 0.3.12** - Base framework
- **langchain-community 0.3.12** - Community integrations
- **qdrant-client 1.12.1** - Vector database client
- **langchain-qdrant 0.2.0** - LangChain + Qdrant integration

### Utilities
- **httpx 0.28.1** - Async HTTP client for external APIs
- **python-dotenv 1.0.1** - Environment configuration
- **structlog** - Structured logging
- **prometheus-client** - Metrics and monitoring

### Container
- **Python 3.12** - Runtime (already upgraded)
- **Docker** - Containerization
- **Network:** ai-dataplane (shared with Ollama, Qdrant, Open WebUI)

## Three-Tier Memory Architecture

### Tier 1: Working Memory (In-Memory)
**Purpose:** Immediate context for ongoing conversation

**Implementation:** `ConversationBufferMemory`
- Stores last 10 conversation turns in RAM
- Fast access (< 1ms)
- Automatic pruning when limit reached
- Lost on container restart (ephemeral)

**Storage:** 0MB persistent, ~5KB RAM

### Tier 2: Short-Term Memory (SQLite)
**Purpose:** Recent conversation summaries

**Implementation:** `ConversationSummaryMemory`
- Summarized conversation history (hours to days)
- Stored in SQLite database
- Medium access speed (~10ms)
- Persists across restarts

**Storage:** `~/docker-data/ai-orchestrator/data/memory.db` (~500KB per 100 conversations)

### Tier 3: Long-Term Memory (Qdrant)
**Purpose:** Semantic search across entire conversation history

**Implementation:** `VectorStoreRetrieverMemory` with Qdrant
- All conversations embedded and stored as vectors
- Semantic similarity search for relevant context
- Unlimited history retention
- Fast semantic search (< 50ms)

**Storage:** Qdrant collection `conversation_memory` (~1KB per turn, 10MB for 10k turns)

### Memory Consolidation Strategy

```python
# Consolidation triggers
CONSOLIDATION_RULES = {
    "message_count": 10,        # Every 10 messages → Summarize to Tier 2
    "token_limit": 2000,        # When context > 2000 tokens → Compress
    "conversation_end": True,   # End of conversation → Embed to Tier 3
    "explicit_save": True,      # User: "remember this" → Force save
}
```

## Multi-Agent Workflow System

### Router Agent (Lightweight Model)
**Model:** gemma:2b or gemma:7b
**Purpose:** Analyze incoming requests and route to appropriate agent/model

**Decision Criteria:**
- Task complexity (token estimation, keyword analysis)
- Domain specialization (code, math, general, creative)
- Tool requirements (web search, file access, image generation)
- Response quality needs (fast vs accurate)

### Chat Agent (General Purpose)
**Model:** mistral:7b (default) or gemma:7b (simple queries)
**Purpose:** Handle general conversation, Q&A, casual interactions

**Capabilities:**
- Normal chat interactions
- Simple questions and answers
- Memory recall from Qdrant
- Basic tool calling (web search, file access)

### Research Agent (Deep Analysis)
**Model:** mixtral:8x7b or mistral:7b
**Purpose:** Complex research tasks requiring web search and synthesis

**Workflow:**
1. Query expansion (generate related search terms)
2. Web search (DuckDuckGo, multiple queries)
3. Content scraping (via Core API)
4. Analysis (extract key information)
5. Synthesis (generate comprehensive report)

**Tools:**
- Web search
- Web scraping (Core API)
- Document retrieval (Qdrant)

### Code Agent (Specialist)
**Model:** codestral:latest or codegemma:latest
**Purpose:** Programming tasks, debugging, code generation

**Capabilities:**
- Code generation (multiple languages)
- Debugging and optimization
- Code explanation
- API integration examples

### Tool Agent (Action Executor)
**Model:** mistral:7b
**Purpose:** Execute actions using external tools and APIs

**Available Tools:**
- **Web Search** (DuckDuckGo)
- **Web Scraping** (Core API)
- **File Operations** (Nextcloud API)
- **Calendar Management** (CalDAV via Nextcloud)
- **Image Generation** (ComfyUI/Stable Diffusion)
- **Home Automation** (Home Assistant - future)
- **Task Management** (future custom system)

## OpenAI-Compatible API Design

### Endpoint: POST /v1/chat/completions

**Request Schema:**
```json
{
  "model": "gpt-3.5-turbo",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Hello!"}
  ],
  "stream": false,
  "temperature": 0.7,
  "max_tokens": 2048
}
```

**Model Aliasing:**
```python
MODEL_ALIASES = {
    "gpt-3.5-turbo": "gemma:7b",           # Fast, lightweight
    "gpt-4": "mistral:7b",                 # High quality
    "gpt-4-turbo": "mixtral:8x7b",         # Very capable
    "gpt-4-code": "codestral:latest",      # Code specialist
    "gpt-4-32k": "gemma2:27b",             # Longer context
}
```

**Response Schema (Non-Streaming):**
```json
{
  "id": "chatcmpl-1234567890",
  "object": "chat.completion",
  "created": 1699564800,
  "model": "gpt-3.5-turbo",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Hello! How can I help you?"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 8,
    "total_tokens": 18
  }
}
```

**Response Schema (Streaming):**
```
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
```

### Additional Endpoints

- `GET /v1/models` - List available models
- `GET /health` - Health check
- `GET /metrics` - Prometheus metrics
- `POST /v1/embeddings` - Generate embeddings (future)

## Tool Integration Plan

### Phase 1 Tools (Core Functionality)

#### 1. Web Search Tool
**Integration:** DuckDuckGo API (no API key required)
**Purpose:** Find current information on the web

```python
class WebSearchTool(BaseTool):
    name = "web_search"
    description = "Search the web for current information"

    def _run(self, query: str, num_results: int = 5) -> str:
        """Execute web search via DuckDuckGo."""
```

#### 2. Web Scraping Tool
**Integration:** Core API (already deployed)
**Purpose:** Extract content from web pages

```python
class WebScrapeTool(BaseTool):
    name = "scrape_webpage"
    description = "Extract clean text content from a URL"

    def _run(self, url: str) -> str:
        """Call Core API scraper endpoint."""
        response = httpx.post(
            "http://core-api:8083/scrape",
            json={"url": url}
        )
```

#### 3. Document Search Tool
**Integration:** Qdrant documents collection
**Purpose:** Search uploaded documents and previous conversations

```python
class DocumentSearchTool(BaseTool):
    name = "search_documents"
    description = "Search through uploaded documents and conversation history"

    def _run(self, query: str) -> str:
        """Semantic search in Qdrant."""
```

### Phase 2 Tools (Productivity)

#### 4. Nextcloud File Tool
**Integration:** Nextcloud WebDAV API
**Purpose:** Search and access files in Nextcloud

```python
class NextcloudFileTool(BaseTool):
    name = "search_files"
    description = "Search for files in Nextcloud"
```

#### 5. Calendar Tool
**Integration:** CalDAV via Nextcloud
**Purpose:** Check calendar, add events

```python
class CalendarTool(BaseTool):
    name = "check_calendar"
    description = "Check calendar for events or add new events"
```

### Phase 3 Tools (Advanced)

#### 6. Image Generation Tool
**Integration:** ComfyUI API (when deployed)
**Purpose:** Generate images from text descriptions

```python
class ImageGenerationTool(BaseTool):
    name = "generate_image"
    description = "Generate images using Stable Diffusion"
```

#### 7. Home Automation Tool
**Integration:** Home Assistant API (when deployed)
**Purpose:** Control smart home devices

```python
class HomeAssistantTool(BaseTool):
    name = "control_home"
    description = "Control smart home devices"
```

#### 8. Task Management Tool
**Integration:** Custom task system (future)
**Purpose:** Create, read, update tasks and reminders

```python
class TaskManagementTool(BaseTool):
    name = "manage_tasks"
    description = "Create and manage tasks and reminders"
```

## Directory Structure

```
/home/jpmschweitzer/Projects/portainer-core/
├── services/
│   └── ai-orchestrator/
│       ├── Dockerfile
│       ├── requirements.txt
│       ├── .env.example
│       ├── README.md
│       └── src/
│           ├── __init__.py
│           ├── main.py              # FastAPI app entry point
│           ├── config.py            # Configuration management
│           │
│           ├── api/                 # API layer
│           │   ├── __init__.py
│           │   ├── routes.py        # Route definitions
│           │   ├── schemas.py       # Pydantic models
│           │   └── middleware.py    # Auth, CORS, logging
│           │
│           ├── agents/              # LangGraph agents
│           │   ├── __init__.py
│           │   ├── router.py        # Router agent
│           │   ├── chat.py          # Chat agent
│           │   ├── research.py      # Research agent
│           │   ├── code.py          # Code agent
│           │   └── tool_executor.py # Tool agent
│           │
│           ├── memory/              # Memory systems
│           │   ├── __init__.py
│           │   ├── working.py       # Tier 1 (in-memory)
│           │   ├── summary.py       # Tier 2 (SQLite)
│           │   ├── vector.py        # Tier 3 (Qdrant)
│           │   └── consolidation.py # Memory consolidation
│           │
│           ├── models/              # Model management
│           │   ├── __init__.py
│           │   ├── router.py        # Model selection logic
│           │   ├── aliases.py       # Model name mapping
│           │   └── manager.py       # Model lifecycle
│           │
│           ├── tools/               # LangChain tools
│           │   ├── __init__.py
│           │   ├── web_search.py    # DuckDuckGo search
│           │   ├── web_scrape.py    # Core API scraper
│           │   ├── document.py      # Qdrant document search
│           │   ├── nextcloud.py     # File operations
│           │   ├── calendar.py      # Calendar management
│           │   ├── image_gen.py     # Image generation
│           │   └── home_assistant.py # Home automation
│           │
│           ├── rag/                 # RAG system
│           │   ├── __init__.py
│           │   ├── retriever.py     # Hybrid retrieval
│           │   ├── embeddings.py    # Embedding generation
│           │   └── reranker.py      # Re-ranking
│           │
│           └── utils/               # Utilities
│               ├── __init__.py
│               ├── logging.py       # Structured logging
│               ├── metrics.py       # Prometheus metrics
│               └── helpers.py       # Common utilities
│
└── stacks/
    └── ai-orchestrator.yml          # Docker Compose stack

/home/jpmschweitzer/docker-data/
└── ai-orchestrator/
    ├── data/
    │   ├── memory.db                # SQLite for summaries
    │   └── checkpoints/             # LangGraph checkpoints
    ├── logs/
    │   └── app.log                  # Application logs
    └── cache/                       # Response cache
```

## Implementation Phases

### Phase 1: Foundation (Week 1) ✅ **COMPLETED 2025-11-13**
**Goal:** Basic OpenAI-compatible API wrapper that works with Open WebUI

**Status:** ✅ All tasks completed. Implementation added to Core API service.

**Tasks:** ✅ **ALL COMPLETE**
1. ✅ Create service directory structure (extended Core API instead)
2. ✅ Implement FastAPI app with `/v1/chat/completions` endpoint
3. ✅ Add OpenAI request/response schemas (Pydantic models)
4. ✅ Connect to Ollama for model inference
5. ✅ Implement basic streaming support (SSE format)
6. ✅ Add model aliasing (gpt-3.5-turbo → gemma:7b)
7. ✅ Create Dockerfile and requirements.txt (reused Core API container)
8. ✅ Create Docker Compose stack definition (updated core-api.yml)
9. ✅ Deploy to ai-dataplane network
10. ✅ Test with Open WebUI

**Deliverables:** ✅ **ALL DELIVERED**
- ✅ Working `/v1/chat/completions` endpoint (src/api/v1/chat.py)
- ✅ Both streaming and non-streaming responses
- ✅ Model name mapping (src/config.py model_aliases)
- ✅ Docker container deployed (core-api on port 8083)
- ✅ `/v1/models` endpoint (src/api/v1/models.py)
- ✅ OllamaClient with connection management (src/models/ollama_client.py)

**Success Criteria:** ✅ **ALL MET**
- ✅ OpenAI-compatible API responding correctly
- ✅ Streaming works properly (Server-Sent Events format)
- ✅ Non-streaming responses working
- ✅ Model aliasing functional (tested gpt-3.5-turbo → gemma:7b)
- ✅ Health check passing, Ollama connectivity verified

**Implementation Notes:**
- Implemented within existing Core API service rather than separate container
- Hot-reload development mode active for rapid iteration
- Ready for Open WebUI integration (endpoint: http://core-api:8083/v1)

### Phase 2: Memory Systems (Week 2)
**Goal:** Persistent conversation memory with three-tier architecture

**Tasks:**
1. Implement Tier 1: ConversationBufferMemory (in-memory)
2. Implement Tier 2: ConversationSummaryMemory (SQLite)
3. Integrate Tier 3: VectorStoreRetrieverMemory (Qdrant)
4. Create Qdrant collections (conversation_memory, documents, user_facts)
5. Implement memory consolidation service
6. Add conversation history API endpoints
7. Build memory recall in conversation flow
8. Test memory persistence across container restarts
9. Add memory metrics (Prometheus)

**Deliverables:**
- Three-tier memory system
- Persistent conversation storage
- Memory consolidation pipeline
- Conversation recall functionality
- Memory metrics dashboard

**Success Criteria:**
- Conversations persist across restarts
- Memory recall provides relevant context
- Semantic search returns appropriate results
- No memory leaks or unbounded growth

### Phase 3: Multi-Agent Workflows (Week 3)
**Goal:** LangGraph-based agent system with intelligent routing

**Tasks:**
1. Install and configure LangGraph
2. Implement Router Agent (analyzes intent, routes requests)
3. Implement Chat Agent (general conversation)
4. Implement Research Agent (multi-step web research)
5. Implement Code Agent (programming specialist)
6. Add agent state management (LangGraph StateGraph)
7. Add supervisor pattern for agent coordination
8. Implement agent selection logic
9. Add agent switching mid-conversation
10. Test complex multi-step workflows

**Deliverables:**
- Working multi-agent system
- Intelligent request routing
- Specialist agent delegation
- Agent state persistence
- Multi-step workflow support

**Success Criteria:**
- Simple queries use lightweight models
- Complex tasks routed to heavy models
- Research tasks trigger multi-step workflows
- Code questions use specialist models
- Agent handoff works seamlessly

### Phase 4: Tool Integration (Week 4)
**Goal:** External API and tool calling capabilities

**Tasks:**
1. Create LangChain tool interface base class
2. Implement Web Search Tool (DuckDuckGo)
3. Implement Web Scrape Tool (Core API integration)
4. Implement Document Search Tool (Qdrant)
5. Test tool calling in agent workflows
6. Add tool usage metrics
7. Implement tool error handling and retries
8. Add tool result caching
9. Create tool documentation
10. Test agent tool usage in real scenarios

**Deliverables:**
- 3 working tools (search, scrape, document)
- Tool calling in agents
- Error handling and retries
- Tool metrics
- Usage documentation

**Success Criteria:**
- Agents can successfully call tools
- Web search returns relevant results
- Web scraping extracts clean content
- Document search finds relevant info
- Tools handle errors gracefully

### Phase 5: RAG & Advanced Memory (Week 5)
**Goal:** Document retrieval and hybrid search

**Tasks:**
1. Implement hybrid retrieval (dense + sparse)
2. Create document embedding pipeline
3. Build RAG chain with Qdrant
4. Add re-ranking for better results
5. Integrate RAG with conversation flow
6. Add document upload endpoint
7. Implement document chunking strategy
8. Test RAG with various document types
9. Optimize retrieval performance
10. Add RAG metrics

**Deliverables:**
- Hybrid search system (semantic + keyword)
- Document embedding pipeline
- RAG-enhanced responses
- Re-ranking optimization
- Document upload API

**Success Criteria:**
- Documents can be uploaded and indexed
- Semantic search returns relevant chunks
- Hybrid search improves accuracy
- RAG responses use document context
- Performance meets targets (< 100ms retrieval)

### Phase 6: Production Hardening (Week 6)
**Goal:** Observability, error handling, optimization

**Tasks:**
1. Add structured logging (structlog)
2. Implement comprehensive error handling
3. Add retry logic for external calls
4. Implement Prometheus metrics
5. Create health check endpoints
6. Add request/response caching
7. Optimize model selection logic
8. Performance testing and optimization
9. Load testing (concurrent requests)
10. Documentation and deployment guide

**Deliverables:**
- Production-ready service
- Monitoring and metrics
- Error handling
- Performance benchmarks
- Load test results
- Complete documentation

**Success Criteria:**
- Structured logs for debugging
- All errors handled gracefully
- Metrics exported to Prometheus
- Health checks pass
- Response times < 2s (p95)
- Can handle 10+ concurrent requests
- Documentation complete

## Docker Configuration

### Dockerfile

```dockerfile
FROM python:3.12-slim

# Prevent Python from writing pyc files and buffering
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONPATH=/app

WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements first (better caching)
COPY requirements.txt .
RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY ./src /app/src

# Create non-root user
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /app
USER appuser

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
    CMD curl -f http://localhost:8084/health || exit 1

# Expose port
EXPOSE 8084

# Run FastAPI with Uvicorn
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8084", "--workers", "1"]
```

### Docker Compose Stack (stacks/ai-orchestrator.yml)

```yaml
version: '3.8'

# AI Orchestrator - LangGraph/LangChain service with OpenAI-compatible API
# Purpose: Intelligent agent workflows with multi-model routing and advanced memory
# Port: 8084 (HTTP API)
# Network: ai-dataplane (shared with Ollama, Qdrant, Open WebUI)
# Dependencies: Ollama (models), Qdrant (memory), Core API (web scraping)

services:
  ai-orchestrator:
    build:
      context: /home/jpmschweitzer/Projects/portainer-core/services/ai-orchestrator
      dockerfile: Dockerfile
    container_name: ai-orchestrator
    restart: unless-stopped

    ports:
      - "8084:8084"

    environment:
      # Application
      - APP_NAME=AI Orchestrator
      - APP_VERSION=1.0.0
      - DEBUG=false
      - LOG_LEVEL=INFO
      - ENVIRONMENT=production

      # Server
      - HOST=0.0.0.0
      - PORT=8084
      - WORKERS=1

      # Model endpoints
      - OLLAMA_BASE_URL=http://ollama:11434
      - QDRANT_URL=http://qdrant:6333
      - CORE_API_URL=http://core-api:8083

      # Model configuration
      - DEFAULT_MODEL=gemma:7b
      - LIGHTWEIGHT_MODELS=gemma:2b,gemma:7b
      - HEAVY_MODELS=mistral:7b,gemma2:9b,mixtral:8x7b
      - CODE_MODELS=codestral:latest,codegemma:latest
      - MATH_MODELS=mistral:7b

      # Model aliases (OpenAI → Local)
      - MODEL_ALIAS_GPT35=gemma:7b
      - MODEL_ALIAS_GPT4=mistral:7b
      - MODEL_ALIAS_GPT4_TURBO=mixtral:8x7b
      - MODEL_ALIAS_GPT4_CODE=codestral:latest

      # Memory configuration
      - MEMORY_COLLECTION=conversation_memory
      - MAX_WORKING_MEMORY=10
      - CONSOLIDATION_INTERVAL=10
      - ENABLE_MEMORY_CONSOLIDATION=true

      # RAG configuration
      - RAG_ENABLED=true
      - EMBEDDING_MODEL=nomic-embed-text
      - RETRIEVAL_K=5
      - HYBRID_SEARCH=true
      - RERANK_ENABLED=true

      # Agent configuration
      - MAX_ITERATIONS=10
      - AGENT_TIMEOUT=300
      - ENABLE_RESEARCH_AGENT=true
      - ENABLE_CODE_AGENT=true
      - ENABLE_TOOL_AGENT=true

      # Tool configuration
      - ENABLE_WEB_SEARCH=true
      - ENABLE_WEB_SCRAPE=true
      - ENABLE_DOCUMENT_SEARCH=true
      - ENABLE_NEXTCLOUD=false
      - ENABLE_CALENDAR=false
      - ENABLE_IMAGE_GEN=false
      - ENABLE_HOME_ASSISTANT=false

      # Performance
      - ENABLE_CACHING=true
      - CACHE_TTL=3600
      - MAX_CONCURRENT_REQUESTS=10

      # Security
      - CORS_ORIGINS=http://192.168.86.149:82,http://open-webui:8080
      - API_KEY_REQUIRED=false
      # - API_KEY=your-secret-key-here

      # Monitoring
      - ENABLE_METRICS=true
      - METRICS_PORT=9090

    volumes:
      # Persistent data (SQLite, checkpoints)
      - /home/jpmschweitzer/docker-data/ai-orchestrator/data:/app/data

      # Logs
      - /home/jpmschweitzer/docker-data/ai-orchestrator/logs:/app/logs

      # Cache
      - /home/jpmschweitzer/docker-data/ai-orchestrator/cache:/app/cache

      # Optional: Mount source for development (hot reload)
      # - /home/jpmschweitzer/Projects/portainer-core/services/ai-orchestrator/src:/app/src

    networks:
      - ai-dataplane

    depends_on:
      - ollama
      - qdrant

    labels:
      - "com.centurylinklabs.watchtower.enable=true"

    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8084/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

networks:
  ai-dataplane:
    external: true

# Deployment Notes:
# 1. Ensure Ollama and Qdrant are running first
# 2. Create data directories: mkdir -p ~/docker-data/ai-orchestrator/{data,logs,cache}
# 3. Deploy in Portainer: Stacks → Add Stack → Upload this file
# 4. Verify health: curl http://localhost:8084/health
# 5. Test API: curl http://localhost:8084/v1/models
# 6. Configure Open WebUI to use this endpoint
```

## Migration Strategy

### Parallel Deployment Approach

The orchestrator will be deployed alongside the existing Ollama connection, allowing gradual migration with easy rollback.

#### Phase 1: Deploy Orchestrator (Week 1)
- Deploy ai-orchestrator container
- Keep Open WebUI pointing to Ollama directly
- Test orchestrator independently using curl/httpx

#### Phase 2: Dual Configuration (Week 2)
- Configure Open WebUI with both endpoints:
  - Primary: Ollama (http://ollama:11434) - Existing
  - Secondary: AI Orchestrator (http://ai-orchestrator:8084/v1) - New

Users can choose which endpoint to use in Open WebUI settings.

#### Phase 3: Gradual Migration (Weeks 3-4)
- Test orchestrator extensively
- Gather user feedback
- Fix issues as they arise
- Demonstrate superior capabilities (memory, tools, research)

#### Phase 4: Primary Switch (Week 5)
- Make orchestrator the default endpoint
- Keep Ollama direct as fallback option
- Monitor for any issues

#### Phase 5: Full Migration (Week 6)
- If stable, make orchestrator the only endpoint
- Document the change
- Keep Ollama direct as admin-only option

### Rollback Plan

If issues arise at any point:
1. Switch Open WebUI back to Ollama direct connection
2. Debug orchestrator issues offline
3. Fix and re-test before re-enabling
4. No downtime for users

### Open WebUI Configuration

**Current Configuration (stacks/open-webui.yml):**
```yaml
environment:
  - OLLAMA_BASE_URL=http://ollama:11434
```

**Dual Configuration (Migration Phase):**
```yaml
environment:
  - OLLAMA_BASE_URL=http://ollama:11434  # Fallback
  - OPENAI_API_BASE=http://ai-orchestrator:8084/v1  # New
  - ENABLE_OPENAI_API=true
```

**Final Configuration (After Migration):**
```yaml
environment:
  - OLLAMA_BASE_URL=http://ai-orchestrator:8084/v1  # Primary
  # - OLLAMA_FALLBACK_URL=http://ollama:11434  # Emergency fallback
```

## Performance Targets

### Response Time Targets

| Scenario | Target (p50) | Target (p95) | Target (p99) |
|----------|--------------|--------------|--------------|
| Simple chat (lightweight model) | < 500ms | < 1s | < 2s |
| Complex chat (heavy model) | < 1s | < 2s | < 4s |
| Research task (multi-step) | < 5s | < 10s | < 15s |
| Tool calling (web search) | < 2s | < 4s | < 6s |
| RAG retrieval | < 100ms | < 200ms | < 500ms |

### Throughput Targets

| Metric | Target |
|--------|--------|
| Concurrent requests | 10+ |
| Requests per minute | 60+ |
| GPU utilization | 60-80% |
| Memory usage (orchestrator) | < 1GB |

### Quality Targets

| Metric | Target |
|--------|--------|
| Model routing accuracy | > 90% |
| Memory recall relevance | > 85% |
| Tool calling success rate | > 95% |
| API compatibility | 100% (OpenAI spec) |

## Monitoring & Observability

### Metrics to Track (Prometheus)

**Request Metrics:**
- `ai_orchestrator_requests_total{model, status}` - Total requests
- `ai_orchestrator_request_duration_seconds{model}` - Request latency histogram
- `ai_orchestrator_active_requests` - Currently processing requests

**Agent Metrics:**
- `ai_orchestrator_agent_invocations_total{agent_type}` - Agent usage
- `ai_orchestrator_agent_duration_seconds{agent_type}` - Agent execution time
- `ai_orchestrator_model_routing_total{from_model, to_model}` - Routing decisions

**Memory Metrics:**
- `ai_orchestrator_memory_consolidations_total` - Memory consolidations
- `ai_orchestrator_memory_retrieval_duration_seconds` - Retrieval time
- `ai_orchestrator_memory_size_bytes{tier}` - Memory size per tier

**Tool Metrics:**
- `ai_orchestrator_tool_calls_total{tool_name, status}` - Tool usage
- `ai_orchestrator_tool_duration_seconds{tool_name}` - Tool execution time
- `ai_orchestrator_tool_errors_total{tool_name}` - Tool failures

### Logging Strategy

**Structured Logging with structlog:**
```python
logger.info(
    "chat_request_received",
    request_id=request_id,
    model=request.model,
    message_count=len(request.messages),
    stream=request.stream,
    user_id=user_id
)
```

**Log Levels:**
- **DEBUG:** Detailed agent workflows, tool calls, memory operations
- **INFO:** Request/response, agent routing decisions
- **WARNING:** Fallbacks, retries, degraded performance
- **ERROR:** Failures, exceptions, unrecoverable errors

### Health Checks

**Endpoint:** `GET /health`

**Checks:**
- API status
- Ollama connectivity
- Qdrant connectivity
- Core API connectivity
- Memory system health
- Disk space

**Response:**
```json
{
  "status": "healthy",
  "timestamp": 1699564800,
  "checks": {
    "api": "healthy",
    "ollama": "healthy",
    "qdrant": "healthy",
    "core_api": "healthy",
    "memory": "healthy",
    "disk": "healthy"
  },
  "version": "1.0.0"
}
```

## Security Considerations

### Authentication (Phase 6)

Optional API key authentication:
```python
@app.post("/v1/chat/completions")
async def chat_completions(
    request: ChatCompletionRequest,
    api_key: str = Depends(verify_api_key)
):
    # ... process request
```

### Rate Limiting (Phase 6)

Prevent abuse:
```python
@limiter.limit("60/minute")  # 60 requests per minute per IP
async def chat_completions(...):
    # ... process request
```

### Input Validation

Pydantic models validate all inputs:
```python
class ChatCompletionRequest(BaseModel):
    model: constr(min_length=1, max_length=100)
    messages: List[ChatMessage]
    max_tokens: Optional[conint(ge=1, le=4096)] = None
```

### CORS Configuration

Restrict origins:
```python
app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://192.168.86.149:82"],  # Open WebUI
    allow_credentials=True,
    allow_methods=["POST", "GET"],
    allow_headers=["*"]
)
```

## Testing Strategy

### Unit Tests
- Test individual components (router, memory, tools)
- Mock external dependencies (Ollama, Qdrant)
- Use pytest and pytest-asyncio

### Integration Tests
- Test complete API endpoints
- Real connections to Ollama/Qdrant
- Test streaming and non-streaming responses

### Load Tests
- Test concurrent request handling
- Measure response times under load
- Identify bottlenecks

### End-to-End Tests
- Test via Open WebUI
- Test complex multi-agent workflows
- Test tool calling and RAG

## Documentation Plan

### User Documentation
- API documentation (OpenAPI/Swagger)
- Model selection guide
- Memory system explanation
- Tool usage examples

### Developer Documentation
- Architecture overview
- Code structure
- Adding new agents
- Adding new tools
- Configuration guide

### Operations Documentation
- Deployment guide
- Monitoring setup
- Troubleshooting guide
- Performance tuning

## Success Criteria

### Phase 1 Success
- ✅ Open WebUI can connect and chat
- ✅ Streaming works correctly
- ✅ Model aliases function
- ✅ No errors in logs

### Phase 2 Success
- ✅ Memory persists across restarts
- ✅ Semantic recall works
- ✅ Consolidation triggers properly
- ✅ No memory leaks

### Phase 3 Success
- ✅ Agent routing works correctly
- ✅ Multi-step workflows complete
- ✅ Specialist agents activate appropriately
- ✅ State management functions

### Phase 4 Success
- ✅ Tools callable from agents
- ✅ Web search returns results
- ✅ Web scraping extracts content
- ✅ Error handling works

### Phase 5 Success
- ✅ Documents indexed and searchable
- ✅ Hybrid search improves results
- ✅ RAG provides relevant context
- ✅ Performance targets met

### Phase 6 Success
- ✅ Metrics exported to Prometheus
- ✅ Health checks pass
- ✅ Load tests successful
- ✅ Documentation complete
- ✅ Production deployment successful

## Future Enhancements (Post-Launch)

### Phase 7+: Advanced Features
1. **Nextcloud Integration** - File search, calendar management
2. **ComfyUI Integration** - Image generation capabilities
3. **Home Assistant Integration** - Smart home control
4. **Task Management System** - Custom task/todo system
5. **Mobile Apps** - Native iOS/Android apps
6. **Home Screen Widgets** - Quick actions and status
7. **Voice Interface** - Voice command processing
8. **Proactive Notifications** - Intelligent reminders
9. **Multi-User Support** - Per-user memory and preferences
10. **Fine-Tuned Models** - Custom models for specific tasks

## Risk Assessment & Mitigation

### Risk 1: GPU VRAM Exhaustion
**Impact:** High - Service fails if VRAM exceeded
**Probability:** Medium - Can happen with concurrent heavy model loads
**Mitigation:**
- Implement model queue (max 1-2 concurrent)
- Use quantized models (Q4, Q5)
- Monitor VRAM usage
- Automatic fallback to CPU for lightweight models

### Risk 2: Qdrant Performance Degradation
**Impact:** Medium - Slower retrieval affects UX
**Probability:** Low - Qdrant is fast with proper indexing
**Mitigation:**
- Use HNSW indexing (default)
- Implement collection partitioning
- Add query filters to reduce search space
- Cache frequent queries

### Risk 3: OpenAI API Incompatibility
**Impact:** High - Open WebUI won't work
**Probability:** Low - Spec is well-defined
**Mitigation:**
- Follow OpenAI API spec exactly
- Test thoroughly with Open WebUI
- Document unsupported features
- Keep Ollama direct as fallback

### Risk 4: Complex Agent Workflows Timeout
**Impact:** Medium - Some tasks fail
**Probability:** Medium - Research tasks can be slow
**Mitigation:**
- Set reasonable timeouts (5 minutes)
- Implement streaming progress updates
- Break down complex tasks
- Return partial results on timeout

### Risk 5: Memory Consolidation Overhead
**Impact:** Low - Slight performance impact
**Probability:** High - Consolidation is CPU intensive
**Mitigation:**
- Run consolidation async (background)
- Batch consolidation operations
- Use lightweight model for summaries
- Monitor consolidation performance

## Cost-Benefit Analysis

### Development Cost
- **Time:** 6 weeks (1 developer)
- **Infrastructure:** $0 (using existing hardware)
- **Opportunity cost:** Medium (could work on other features)

### Benefits
- **Superior Memory:** Proper conversation context and recall
- **Multi-Agent Intelligence:** Right model for each task
- **Tool Integration:** Web search, file access, automation
- **Research Capabilities:** Deep information gathering
- **Future-Proof:** Foundation for mobile apps and custom UI
- **Better UX:** Faster, more accurate, more capable

### ROI
- **Short-term:** Improved AI interactions immediately
- **Medium-term:** Platform for advanced features
- **Long-term:** Foundation for custom AI applications

## Conclusion

This implementation plan provides a comprehensive roadmap for building a sophisticated AI orchestration layer that transforms the tower-of-joy infrastructure from a basic LLM chat interface into an intelligent, multi-agent system with proper memory, tool integration, and research capabilities.

The phased approach ensures steady progress with testable milestones, while the parallel deployment strategy minimizes risk and allows for easy rollback if needed. The architecture is designed to integrate seamlessly with existing infrastructure (Ollama, Qdrant, Core API) while providing a foundation for future enhancements like mobile apps and home automation.

By the end of Week 6, the system will provide:
- ✅ OpenAI-compatible API for Open WebUI
- ✅ Three-tier memory system with semantic recall
- ✅ Multi-agent workflows with intelligent routing
- ✅ Tool integration (web search, scraping, documents)
- ✅ RAG with hybrid search
- ✅ Production-grade monitoring and observability

This establishes the tower-of-joy project as a cutting-edge AI homelab with capabilities rivaling commercial solutions, all running on local hardware with full data sovereignty.

---

**Plan Status:** ✅ Phase 1 Complete - 🔄 Phase 2 In Progress
**Completed:** Phase 1 - Foundation (2025-11-13)
**Next Step:** Phase 2 - Memory Systems (Week 2)