diff --git a/plans/active/ai-orchestrator-plan.md b/plans/active/ai-orchestrator-plan.md deleted file mode 100644 index 10e7a7f..0000000 --- a/plans/active/ai-orchestrator-plan.md +++ /dev/null @@ -1,1102 +0,0 @@ -# AI Orchestrator Implementation Plan - -> **Project:** tower-of-joy AI Stack Enhancement -> **Created:** 2025-11-13 -> **Status:** Phase 4 Complete ✅ - PydanticAI Migration Successful -> **Updated:** 2025-12-03 -> **Framework:** PydanticAI with Ollama - -## Executive Summary - -This document outlines the implementation of a sophisticated AI orchestration layer using PydanticAI and aiohttp that provides an advanced agent system for the tower-of-joy homelab. The architecture provides: - -**✅ MIGRATION COMPLETE (2025-11-26):** -Successfully migrated from LangChain/LangGraph to PydanticAI to achieve reliable tool calling with Ollama local models. - -- **Advanced Memory Systems:** Three-tier memory with Qdrant for long-term semantic recall -- **Multi-Agent Workflows:** Intelligent routing to lightweight, heavy, and specialist models -- **Extensive Tool Integration:** Web search, file operations, calendar, home automation, image generation -- **Production-Ready API:** OpenAI-compatible endpoints for seamless Open WebUI integration -- **Superior Performance:** Proper context management, caching, and model selection - -## Current vs Target Architecture - -### Current Architecture (v0.6.0) - -``` -┌──────────────┐ -│ Open WebUI │ -│ (Port 82) │ -└──────┬───────┘ - │ - │ Direct connection - │ -┌──────▼───────┐ ┌──────────────┐ -│ Ollama │ │ Qdrant │ -│ (Port 11434)│ │ (Port 6333) │ -└──────────────┘ └──────────────┘ - │ - │ GPU inference - │ -┌──────▼───────┐ -│ RTX 2080 Ti │ -│ (11GB) │ -└──────────────┘ -``` - -**Limitations:** -- Open WebUI's memory integration not working well -- No intelligent model routing -- Limited tool calling capabilities -- Single-model processing (no multi-agent coordination) -- Difficult to customize RAG behavior - -### Target Architecture - -``` -┌────────────────────────────────────────────────────────────────┐ -│ User Interface Layer │ -│ Open WebUI (Port 82) │ -└───────────────────────────┬────────────────────────────────────┘ - │ - │ /v1/chat/completions (OpenAI-compatible) - │ -┌───────────────────────────▼────────────────────────────────────┐ -│ AI Orchestrator (Port 8084) │ -│ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │ -│ ┃ aiohttp + PydanticAI Orchestration Layer ┃ │ -│ ┃ • OpenAI-compatible API wrapper ┃ │ -│ ┃ • Request routing & agent coordination ┃ │ -│ ┃ • Memory management (3-tier system) ┃ │ -│ ┗━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ │ -│ │ │ -│ ┌──────────────────────┼──────────────────────┐ │ -│ │ │ │ │ -│ ┌─▼──────────┐ ┌───────▼───────┐ ┌─────────▼──────┐ │ -│ │ Chat Agent │ │ Research Agent│ │ Tool Agent │ │ -│ │ (General) │ │ (Deep search) │ │ (Actions) │ │ -│ └─────┬──────┘ └───────┬───────┘ └─────────┬──────┘ │ -└────────┼──────────────────┼─────────────────────┼─────────────┘ - │ │ │ - │ │ │ -┌────────▼──────────────────▼─────────────────────▼─────────────┐ -│ Model Inference Layer (Ollama) │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ -│ │ Lightweight │ │ Heavy │ │ Specialist │ │ -│ │ gemma:2b │ │ mistral:7b │ │ codestral │ │ -│ │ gemma:7b │ │ gemma2:9b │ │ codegemma │ │ -│ │ │ │ gemma2:27b │ │ mixtral:8x7b │ │ -│ │ ~2-4GB VRAM │ │ ~6-8GB VRAM │ │ ~8-10GB VRAM │ │ -│ └──────────────┘ └──────────────┘ └──────────────┘ │ -└───────────────────────────────────────────────────────────────┘ - │ - │ GPU acceleration - │ -┌────────▼───────────────────────────────────────────────────────┐ -│ NVIDIA RTX 2080 Ti (11GB VRAM) │ -└─────────────────────────────────────────────────────────────────┘ - -┌─────────────────────────┐ ┌───────────────────────────────┐ -│ Memory & Storage │ │ External Tools & APIs │ -│ (Qdrant Port 6333) │ │ │ -│ │ │ • Core API (web scraper) │ -│ Tier 1: Working Memory │ │ • Nextcloud API (files) │ -│ Tier 2: Summaries │ │ • Calendar (CalDAV) │ -│ Tier 3: Vector Store │ │ • ComfyUI (image gen) │ -│ • Conversations │ │ • Home Assistant (IoT) │ -│ • Documents │ │ • Web Search (DuckDuckGo) │ -│ • User facts │ │ • Future integrations │ -└─────────────────────────┘ └───────────────────────────────┘ -``` - -**Advantages:** -- ✅ Intelligent multi-model routing (right model for the task) -- ✅ Proper conversation memory with semantic recall -- ✅ Multi-agent coordination for complex tasks -- ✅ Extensive tool calling (web search, files, calendar, automation) -- ✅ Research agents for deep information gathering -- ✅ Image generation via ComfyUI integration -- ✅ Gradual migration path (run parallel with current setup) -- ✅ Foundation for custom mobile apps later - -## Technology Stack - -### Core Framework (Current - Post-Migration) -- **PydanticAI** - Modern AI agent framework with type safety -- **Ollama Native SDK ≥0.4.0** - Direct Ollama integration with tool calling -- **aiohttp 3.10.1** - Async HTTP server framework -- **aiohttp-cors 0.7.0** - CORS support for web clients -- **Pydantic ≥2.10.3** - Request/response validation and type safety -- **pydantic-settings 2.6.1** - Configuration management - -### AI & Memory -- **pydantic-ai-slim** - Minimal PydanticAI library -- **qdrant-client ≥1.12.0** - Vector database client -- **Ollama** - Local LLM inference (native integration) - -### Utilities -- **httpx ≥0.28.0** - Async HTTP client for external APIs -- **python-dotenv ≥1.1.0** - Environment configuration -- **pytz ≥2025.2** - Timezone support for date/time tools - -### Container -- **Python 3.12** - Runtime -- **Docker** - Containerization -- **Network:** ai-dataplane (shared with Ollama, Qdrant, Open WebUI) - -## Three-Tier Memory Architecture - -### Tier 1: Working Memory (In-Memory) -**Purpose:** Immediate context for ongoing conversation - -**Implementation:** `ConversationBufferMemory` -- Stores last 10 conversation turns in RAM -- Fast access (< 1ms) -- Automatic pruning when limit reached -- Lost on container restart (ephemeral) - -**Storage:** 0MB persistent, ~5KB RAM - -### Tier 2: Short-Term Memory (SQLite) -**Purpose:** Recent conversation summaries - -**Implementation:** `ConversationSummaryMemory` -- Summarized conversation history (hours to days) -- Stored in SQLite database -- Medium access speed (~10ms) -- Persists across restarts - -**Storage:** `~/docker-data/ai-orchestrator/data/memory.db` (~500KB per 100 conversations) - -### Tier 3: Long-Term Memory (Qdrant) -**Purpose:** Semantic search across entire conversation history - -**Implementation:** `VectorStoreRetrieverMemory` with Qdrant -- All conversations embedded and stored as vectors -- Semantic similarity search for relevant context -- Unlimited history retention -- Fast semantic search (< 50ms) - -**Storage:** Qdrant collection `conversation_memory` (~1KB per turn, 10MB for 10k turns) - -### Memory Consolidation Strategy - -```python -# Consolidation triggers -CONSOLIDATION_RULES = { - "message_count": 10, # Every 10 messages → Summarize to Tier 2 - "token_limit": 2000, # When context > 2000 tokens → Compress - "conversation_end": True, # End of conversation → Embed to Tier 3 - "explicit_save": True, # User: "remember this" → Force save -} -``` - -## Multi-Agent Workflow System - -### Router Agent (Lightweight Model) -**Model:** gemma:2b or gemma:7b -**Purpose:** Analyze incoming requests and route to appropriate agent/model - -**Decision Criteria:** -- Task complexity (token estimation, keyword analysis) -- Domain specialization (code, math, general, creative) -- Tool requirements (web search, file access, image generation) -- Response quality needs (fast vs accurate) - -### Chat Agent (General Purpose) -**Model:** mistral:7b (default) or gemma:7b (simple queries) -**Purpose:** Handle general conversation, Q&A, casual interactions - -**Capabilities:** -- Normal chat interactions -- Simple questions and answers -- Memory recall from Qdrant -- Basic tool calling (web search, file access) - -### Research Agent (Deep Analysis) -**Model:** mixtral:8x7b or mistral:7b -**Purpose:** Complex research tasks requiring web search and synthesis - -**Workflow:** -1. Query expansion (generate related search terms) -2. Web search (DuckDuckGo, multiple queries) -3. Content scraping (via Core API) -4. Analysis (extract key information) -5. Synthesis (generate comprehensive report) - -**Tools:** -- Web search -- Web scraping (Core API) -- Document retrieval (Qdrant) - -### Code Agent (Specialist) -**Model:** codestral:latest or codegemma:latest -**Purpose:** Programming tasks, debugging, code generation - -**Capabilities:** -- Code generation (multiple languages) -- Debugging and optimization -- Code explanation -- API integration examples - -### Tool Agent (Action Executor) -**Model:** mistral:7b -**Purpose:** Execute actions using external tools and APIs - -**Available Tools:** -- **Web Search** (DuckDuckGo) -- **Web Scraping** (Core API) -- **File Operations** (Nextcloud API) -- **Calendar Management** (CalDAV via Nextcloud) -- **Image Generation** (ComfyUI/Stable Diffusion) -- **Home Automation** (Home Assistant - future) -- **Task Management** (future custom system) - -## OpenAI-Compatible API Design - -### Endpoint: POST /v1/chat/completions - -**Request Schema:** -```json -{ - "model": "gpt-3.5-turbo", - "messages": [ - {"role": "system", "content": "You are a helpful assistant"}, - {"role": "user", "content": "Hello!"} - ], - "stream": false, - "temperature": 0.7, - "max_tokens": 2048 -} -``` - -**Model Aliasing:** -```python -MODEL_ALIASES = { - "gpt-3.5-turbo": "gemma:7b", # Fast, lightweight - "gpt-4": "mistral:7b", # High quality - "gpt-4-turbo": "mixtral:8x7b", # Very capable - "gpt-4-code": "codestral:latest", # Code specialist - "gpt-4-32k": "gemma2:27b", # Longer context -} -``` - -**Response Schema (Non-Streaming):** -```json -{ - "id": "chatcmpl-1234567890", - "object": "chat.completion", - "created": 1699564800, - "model": "gpt-3.5-turbo", - "choices": [{ - "index": 0, - "message": { - "role": "assistant", - "content": "Hello! How can I help you?" - }, - "finish_reason": "stop" - }], - "usage": { - "prompt_tokens": 10, - "completion_tokens": 8, - "total_tokens": 18 - } -} -``` - -**Response Schema (Streaming):** -``` -data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]} - -data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]} - -data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} - -data: [DONE] -``` - -### Additional Endpoints - -- `GET /v1/models` - List available models -- `GET /health` - Health check -- `GET /metrics` - Prometheus metrics -- `POST /v1/embeddings` - Generate embeddings (future) - -## Tool Integration Plan - -### Phase 1 Tools (Core Functionality) - -#### 1. Web Search Tool -**Integration:** DuckDuckGo API (no API key required) -**Purpose:** Find current information on the web - -```python -async def web_search(query: str, num_results: int = 5) -> str: - """Search the web for current information.""" - # Implementation using duckduckgo-search -``` - -#### 2. Web Scraping Tool -**Integration:** Core API (already deployed) -**Purpose:** Extract content from web pages - -```python -async def scrape_webpage(url: str) -> str: - """Extract clean text content from a URL.""" - response = await httpx.post( - "http://core-api:8083/scrape", - json={"url": url} - ) -``` - -#### 3. Document Search Tool -**Integration:** Qdrant documents collection -**Purpose:** Search uploaded documents and previous conversations - -```python -async def search_documents(query: str) -> str: - """Search through uploaded documents and conversation history.""" - # Semantic search in Qdrant -``` - -### Phase 2 Tools (Productivity) - -#### 4. Nextcloud File Tool -**Integration:** Nextcloud WebDAV API -**Purpose:** Search and access files in Nextcloud - -```python -async def search_files(query: str) -> str: - """Search for files in Nextcloud.""" -``` - -#### 5. Calendar Tool -**Integration:** CalDAV via Nextcloud -**Purpose:** Check calendar, add events - -```python -async def check_calendar(date_range: str = "today") -> str: - """Check calendar for events or add new events.""" -``` - -### Phase 3 Tools (Advanced) - -#### 6. Image Generation Tool -**Integration:** ComfyUI API (when deployed) -**Purpose:** Generate images from text descriptions - -```python -async def generate_image(description: str) -> str: - """Generate images using Stable Diffusion.""" -``` - -#### 7. Home Automation Tool -**Integration:** Home Assistant API (when deployed) -**Purpose:** Control smart home devices - -```python -async def control_home(device: str, action: str) -> str: - """Control smart home devices.""" -``` - -#### 8. Task Management Tool -**Integration:** Custom task system (future) -**Purpose:** Create, read, update tasks and reminders - -```python -async def manage_tasks(action: str, task: str = None) -> str: - """Create and manage tasks and reminders.""" -``` - -## Directory Structure - -``` -/home/jpmschweitzer/Projects/portainer-core/ -├── services/ -│ └── core-ai/ -│ ├── Dockerfile -│ ├── requirements.txt -│ ├── .env.example -│ ├── README.md -│ ├── main.py # aiohttp app entry point -│ └── src/ -│ ├── __init__.py -│ ├── config.py # Configuration management -│ ├── prompts.py # System prompt variants -│ ├── utils.py # Common utilities -│ │ -│ ├── agents/ # Agent implementations -│ │ ├── __init__.py -│ │ ├── pydantic_agent.py # Main PydanticAI agent -│ │ ├── simple.py # Fallback LiteLLM agent -│ │ └── ollama_native_agent.py # Native Ollama agent -│ │ -│ ├── memory/ # Memory systems -│ │ ├── __init__.py -│ │ ├── manager.py # Memory manager -│ │ ├── types.py # Memory types and enums -│ │ └── consolidation.py # Memory consolidation -│ │ -│ ├── tools/ # Tool implementations -│ │ ├── __init__.py -│ │ ├── local.py # Local tools (time, services) -│ │ ├── registry.py # Tool registry -│ │ └── openapi_discovery.py # OpenAPI tool discovery -│ │ -│ └── tests/ # Test suite -│ ├── __init__.py -│ ├── test_ai_flow_quality.py -│ └── reports/ -│ -└── stacks/ - └── core-ai.yml # Docker Compose stack - -/home/jpmschweitzer/docker-data/ -└── core-ai/ - ├── data/ - │ ├── memory.db # SQLite for summaries - │ └── checkpoints/ # Agent checkpoints - ├── logs/ - │ └── app.log # Application logs - └── cache/ # Response cache -``` - -## Implementation Phases - -### Phase 1: Foundation (Week 1) ✅ **COMPLETED 2025-11-13** -**Goal:** Basic OpenAI-compatible API wrapper that works with Open WebUI - -**Status:** ✅ All tasks completed. Implementation added to Core API service. - -**Tasks:** ✅ **ALL COMPLETE** -1. ✅ Create service directory structure (extended Core API instead) -2. ✅ Implement aiohttp app with `/v1/chat/completions` endpoint -3. ✅ Add OpenAI request/response schemas (Pydantic models) -4. ✅ Connect to Ollama for model inference -5. ✅ Implement basic streaming support (SSE format) -6. ✅ Add model aliasing (gpt-3.5-turbo → gemma:7b) -7. ✅ Create Dockerfile and requirements.txt -8. ✅ Create Docker Compose stack definition -9. ✅ Deploy to ai-dataplane network -10. ✅ Test with Open WebUI - -**Deliverables:** ✅ **ALL DELIVERED** -- ✅ Working `/v1/chat/completions` endpoint -- ✅ Both streaming and non-streaming responses -- ✅ Model name mapping -- ✅ Docker container deployed (core-ai on port 8084) -- ✅ `/v1/models` endpoint -- ✅ Ollama client with connection management - -**Success Criteria:** ✅ **ALL MET** -- ✅ OpenAI-compatible API responding correctly -- ✅ Streaming works properly (Server-Sent Events format) -- ✅ Non-streaming responses working -- ✅ Model aliasing functional -- ✅ Health check passing, Ollama connectivity verified - -### Phase 2: Memory Systems (Week 2) ✅ **COMPLETED 2025-11-23** -**Goal:** Persistent conversation memory with three-tier architecture - -**Status:** ✅ Complete. Three-tier memory system implemented. - -**Completed Tasks:** -1. ✅ Implement Tier 1: ConversationBufferMemory (in-memory) -2. ✅ Implement Tier 2: ConversationSummaryMemory (SQLite) -3. ✅ Integrate Tier 3: VectorStoreRetrieverMemory (Qdrant) -4. ✅ Create Qdrant collections (conversation_memory, documents, user_facts) -5. ✅ Implement memory consolidation service -6. ✅ Add conversation history API endpoints -7. ✅ Build memory recall in conversation flow -8. ✅ Test memory persistence across container restarts -9. ✅ Add memory metrics - -**Deliverables:** ✅ **ALL DELIVERED** -- ✅ Three-tier memory system -- ✅ Persistent conversation storage -- ✅ Memory consolidation pipeline -- ✅ Conversation recall functionality -- ✅ Memory metrics - -**Success Criteria:** ✅ **ALL MET** -- ✅ Conversations persist across restarts -- ✅ Memory recall provides relevant context -- ✅ Semantic search returns appropriate results -- ✅ No memory leaks or unbounded growth - -### Phase 3: Research Capabilities ✅ **COMPLETE** (2025-11-24) -**Goal:** Web search and research workflows - -**Status:** ✅ **COMPLETE** - All success criteria met - -**Completed Tasks:** -1. ✅ Integrated DuckDuckGo web search API -2. ✅ Implemented automatic content scraping from search results -3. ✅ Added web_search tool (search + scrape in one call) -4. ✅ Added web_scrape tool (targeted URL extraction) -5. ✅ Enhanced system prompts for research query detection -6. ✅ Added enhanced progress indicators -7. ✅ Comprehensive testing (100% success rate) -8. ✅ Tool invocation logging for debugging -9. ✅ System prompt optimization -10. ✅ Model validation (mistral:7b confirmed best for tools) - -**Deliverables:** ✅ **ALL DELIVERED** -- ✅ Web search with DuckDuckGo integration -- ✅ Automatic content extraction from results -- ✅ Research query detection in system prompts -- ✅ Multiple tools (infrastructure + research) -- ✅ Enhanced user experience with visual feedback -- ✅ Comprehensive test suite - -**Success Criteria:** ✅ **ALL MET** -- ✅ Research detection accuracy: 100% -- ✅ Average response time: 5.3s (target: <10s) -- ✅ Source citation rate: 100% -- ✅ Tool calling reliability: 100% on complex queries -- ✅ No regression in existing functionality - -### Phase 4: Framework Migration ✅ **COMPLETE** (2025-11-26) -**Goal:** Migrate from LangChain/LangGraph to PydanticAI for improved reliability - -**Status:** ✅ **COMPLETE** - All success criteria met - -**Completed Tasks:** -1. ✅ Updated requirements.txt (removed langchain*, added pydantic-ai) -2. ✅ Rewrote agent to use PydanticAI with native Ollama SDK -3. ✅ Converted all tools to PydanticAI format -4. ✅ Updated streaming for PydanticAI event format -5. ✅ Optimized model selection (mistral-tools:7b) -6. ✅ Created PydanticAI-optimized system prompts -7. ✅ Enhanced health checks for agent monitoring -8. ✅ Production testing and validation -9. ✅ Documentation updates - -**Deliverables:** ✅ **ALL DELIVERED** -- ✅ PydanticAI integration with native Ollama SDK -- ✅ All tools migrated to PydanticAI format -- ✅ Model optimized for tool calling -- ✅ PydanticAI-optimized system prompts -- ✅ Improved streaming consistency -- ✅ Enhanced agent health monitoring -- ✅ Complete documentation - -**Success Criteria:** ✅ **ALL MET** -- ✅ Tool calling works reliably -- ✅ All models supported (Gemma, Mistral, etc.) -- ✅ Streaming output consistent and clean -- ✅ No regressions in memory system -- ✅ API endpoints unchanged (backward compatible) -- ✅ Performance within targets - -**Implementation Details:** -- **Framework:** PydanticAI with native Ollama SDK -- **Model:** mistral-tools:7b (optimized for tool calling) -- **Prompt:** Optimized variants for PydanticAI -- **Tools:** All tools as PydanticAI-compatible functions -- **Migration Time:** ~6 hours (as estimated) - -**Performance (Post-Migration):** -- Simple queries: ~0.3-1s -- Tool-using queries: ~2-5s (improved from LangChain) -- Research queries: ~4-7s (maintained) -- VRAM usage: ~4-6GB with mistral-tools:7b - -**Why This Migration:** -- ❌ **Problem:** LangChain/LangGraph failed to trigger tools reliably with Ollama -- ❌ **Problem:** Gemma models returned errors with LangChain -- ❌ **Problem:** Inconsistent streaming behavior -- ✅ **Solution:** PydanticAI has native Ollama support with excellent type safety -- ✅ **Result:** Reliable tool calling across all models - -### Phase 5: Multi-Agent Patterns & RAG Optimization (Future) -**Goal:** Enhanced multi-agent workflows and advanced RAG capabilities - -**Tasks:** -1. Implement router agent pattern for intelligent task routing -2. Create specialized agents (code, research, tool execution) -3. Build agent coordination system -4. Implement hybrid retrieval (dense + sparse) -5. Create document embedding pipeline -6. Build RAG chain with Qdrant -7. Add re-ranking for better results -8. Integrate RAG with conversation flow -9. Add document upload endpoint -10. Optimize retrieval performance - -**Deliverables:** -- Multi-agent routing system -- Specialized agent implementations -- Hybrid search system (semantic + keyword) -- Document embedding pipeline -- RAG-enhanced responses -- Re-ranking optimization -- Document upload API - -**Success Criteria:** -- Router correctly selects appropriate agent -- Specialized agents outperform general agent -- Documents can be uploaded and indexed -- Semantic search returns relevant chunks -- Hybrid search improves accuracy -- RAG responses use document context -- Performance meets targets (< 100ms retrieval) - -### Phase 6: Production Hardening (Future) -**Goal:** Observability, error handling, optimization - -**Tasks:** -1. Add structured logging (structlog) -2. Implement comprehensive error handling -3. Add retry logic for external calls -4. Implement Prometheus metrics -5. Create health check endpoints -6. Add request/response caching -7. Optimize model selection logic -8. Performance testing and optimization -9. Load testing (concurrent requests) -10. Documentation and deployment guide - -**Deliverables:** -- Production-ready service -- Monitoring and metrics -- Error handling -- Performance benchmarks -- Load test results -- Complete documentation - -**Success Criteria:** -- Structured logs for debugging -- All errors handled gracefully -- Metrics exported to Prometheus -- Health checks pass -- Response times < 2s (p95) -- Can handle 10+ concurrent requests -- Documentation complete - -## Docker Configuration - -### Dockerfile - -```dockerfile -FROM python:3.12-slim - -# Prevent Python from writing pyc files and buffering -ENV PYTHONDONTWRITEBYTECODE=1 \ - PYTHONUNBUFFERED=1 \ - PYTHONPATH=/app - -WORKDIR /app - -# Install system dependencies -RUN apt-get update && apt-get install -y \ - curl \ - && rm -rf /var/lib/apt/lists/* - -# Copy requirements first (better caching) -COPY requirements.txt . -RUN pip install --no-cache-dir --upgrade pip && \ - pip install --no-cache-dir -r requirements.txt - -# Copy application code -COPY ./src /app/src -COPY ./main.py /app/ - -# Create non-root user -RUN useradd -m -u 1000 appuser && \ - chown -R appuser:appuser /app -USER appuser - -# Health check -HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \ - CMD curl -f http://localhost:8084/health || exit 1 - -# Expose port -EXPOSE 8084 - -# Run aiohttp server -CMD ["python", "main.py"] -``` - -### Docker Compose Stack (stacks/core-ai.yml) - -```yaml -version: '3.8' - -# Core AI - PydanticAI service with OpenAI-compatible API -# Purpose: Intelligent agent workflows with multi-model routing and advanced memory -# Port: 8084 (HTTP API) -# Network: ai-dataplane (shared with Ollama, Qdrant, Open WebUI) -# Dependencies: Ollama (models), Qdrant (memory) - -services: - core-ai: - build: - context: /home/jpmschweitzer/Projects/portainer-core/services/core-ai - dockerfile: Dockerfile - container_name: core-ai - restart: unless-stopped - - ports: - - "8084:8084" - - environment: - # Application - - APP_NAME=Core AI - - APP_VERSION=1.0.0 - - DEBUG=false - - LOG_LEVEL=INFO - - ENVIRONMENT=production - - # Server - - HOST=0.0.0.0 - - PORT=8084 - - # Model endpoints - - OLLAMA_BASE_URL=http://ollama:11434 - - QDRANT_URL=http://qdrant:6333 - - # Model configuration - - DEFAULT_MODEL=mistral-tools:7b - - AGENT_MODEL=mistral-tools:7b - - # Memory configuration - - MEMORY_ENABLED=true - - MEMORY_TIER1_SIZE=10 - - DEFAULT_USER_ID=default - - # Agent configuration - - ENABLE_TOOLS=true - - SYSTEM_PROMPT_VARIANT=minimal_agent - - volumes: - # Persistent data (SQLite, checkpoints) - - /home/jpmschweitzer/docker-data/core-ai/data:/app/data - - # Logs - - /home/jpmschweitzer/docker-data/core-ai/logs:/app/logs - - # Cache - - /home/jpmschweitzer/docker-data/core-ai/cache:/app/cache - - networks: - - ai-dataplane - - depends_on: - - ollama - - qdrant - - labels: - - "com.centurylinklabs.watchtower.enable=true" - - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8084/health"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 30s - -networks: - ai-dataplane: - external: true -``` - -## Performance Targets - -### Response Time Targets - -| Scenario | Target (p50) | Target (p95) | Target (p99) | -|----------|--------------|--------------|--------------| -| Simple chat (lightweight model) | < 500ms | < 1s | < 2s | -| Complex chat (heavy model) | < 1s | < 2s | < 4s | -| Research task (multi-step) | < 5s | < 10s | < 15s | -| Tool calling (web search) | < 2s | < 4s | < 6s | -| RAG retrieval | < 100ms | < 200ms | < 500ms | - -### Throughput Targets - -| Metric | Target | -|--------|--------| -| Concurrent requests | 10+ | -| Requests per minute | 60+ | -| GPU utilization | 60-80% | -| Memory usage (orchestrator) | < 1GB | - -### Quality Targets - -| Metric | Target | -|--------|--------| -| Model routing accuracy | > 90% | -| Memory recall relevance | > 85% | -| Tool calling success rate | > 95% | -| API compatibility | 100% (OpenAI spec) | - -## Monitoring & Observability - -### Metrics to Track (Prometheus) - -**Request Metrics:** -- `core_ai_requests_total{model, status}` - Total requests -- `core_ai_request_duration_seconds{model}` - Request latency histogram -- `core_ai_active_requests` - Currently processing requests - -**Agent Metrics:** -- `core_ai_agent_invocations_total{agent_type}` - Agent usage -- `core_ai_agent_duration_seconds{agent_type}` - Agent execution time -- `core_ai_model_routing_total{from_model, to_model}` - Routing decisions - -**Memory Metrics:** -- `core_ai_memory_consolidations_total` - Memory consolidations -- `core_ai_memory_retrieval_duration_seconds` - Retrieval time -- `core_ai_memory_size_bytes{tier}` - Memory size per tier - -**Tool Metrics:** -- `core_ai_tool_calls_total{tool_name, status}` - Tool usage -- `core_ai_tool_duration_seconds{tool_name}` - Tool execution time -- `core_ai_tool_errors_total{tool_name}` - Tool failures - -### Logging Strategy - -**Structured Logging:** -```python -logger.info( - "chat_request_received", - request_id=request_id, - model=request.model, - message_count=len(request.messages), - stream=request.stream, - user_id=user_id -) -``` - -**Log Levels:** -- **DEBUG:** Detailed agent workflows, tool calls, memory operations -- **INFO:** Request/response, agent routing decisions -- **WARNING:** Fallbacks, retries, degraded performance -- **ERROR:** Failures, exceptions, unrecoverable errors - -### Health Checks - -**Endpoint:** `GET /health` - -**Checks:** -- API status -- Ollama connectivity -- Qdrant connectivity -- Memory system health -- Disk space - -**Response:** -```json -{ - "status": "healthy", - "timestamp": 1699564800, - "checks": { - "api": "healthy", - "ollama": "healthy", - "qdrant": "healthy", - "memory": "healthy", - "disk": "healthy" - }, - "version": "1.0.0" -} -``` - -## Security Considerations - -### Authentication (Phase 6) - -Optional API key authentication: -```python -async def verify_api_key(request): - api_key = request.headers.get("Authorization") - if not api_key: - raise web.HTTPUnauthorized() - # Verify API key -``` - -### Rate Limiting (Phase 6) - -Prevent abuse: -```python -# Implement rate limiting middleware -# 60 requests per minute per IP -``` - -### Input Validation - -Pydantic models validate all inputs: -```python -class ChatCompletionRequest(BaseModel): - model: str = Field(min_length=1, max_length=100) - messages: List[ChatMessage] - max_tokens: Optional[int] = Field(ge=1, le=4096, default=None) -``` - -### CORS Configuration - -Restrict origins: -```python -cors = aiohttp_cors.setup(app, defaults={ - "http://192.168.86.149:82": aiohttp_cors.ResourceOptions( - allow_credentials=True, - expose_headers="*", - allow_headers="*", - ) -}) -``` - -## Testing Strategy - -### Unit Tests -- Test individual components (agents, memory, tools) -- Mock external dependencies (Ollama, Qdrant) -- Use pytest and pytest-asyncio - -### Integration Tests -- Test complete API endpoints -- Real connections to Ollama/Qdrant -- Test streaming and non-streaming responses - -### Load Tests -- Test concurrent request handling -- Measure response times under load -- Identify bottlenecks - -### End-to-End Tests -- Test via Open WebUI -- Test complex multi-agent workflows -- Test tool calling and RAG - -## Success Criteria - -### Phase 1 Success ✅ -- ✅ Open WebUI can connect and chat -- ✅ Streaming works correctly -- ✅ Model aliases function -- ✅ No errors in logs - -### Phase 2 Success ✅ -- ✅ Memory persists across restarts -- ✅ Semantic recall works -- ✅ Consolidation triggers properly -- ✅ No memory leaks - -### Phase 3 Success ✅ -- ✅ Web search functional -- ✅ Content extraction works -- ✅ Research query detection accurate -- ✅ Tool calling reliable - -### Phase 4 Success ✅ -- ✅ PydanticAI integration complete -- ✅ Tools callable from agents -- ✅ Streaming works properly -- ✅ No regressions - -### Phase 5 Success (Future) -- Multi-agent routing functional -- Specialized agents outperform general agent -- Documents indexed and searchable -- Hybrid search improves results -- RAG provides relevant context -- Performance targets met - -### Phase 6 Success (Future) -- Metrics exported to Prometheus -- Health checks pass -- Load tests successful -- Documentation complete -- Production deployment successful - -## Future Enhancements (Post-Launch) - -### Phase 7+: Advanced Features -1. **Nextcloud Integration** - File search, calendar management -2. **ComfyUI Integration** - Image generation capabilities -3. **Home Assistant Integration** - Smart home control -4. **Task Management System** - Custom task/todo system -5. **Mobile Apps** - Native iOS/Android apps -6. **Home Screen Widgets** - Quick actions and status -7. **Voice Interface** - Voice command processing -8. **Proactive Notifications** - Intelligent reminders -9. **Multi-User Support** - Per-user memory and preferences -10. **Fine-Tuned Models** - Custom models for specific tasks - -## Risk Assessment & Mitigation - -### Risk 1: GPU VRAM Exhaustion -**Impact:** High - Service fails if VRAM exceeded -**Probability:** Medium - Can happen with concurrent heavy model loads -**Mitigation:** -- Implement model queue (max 1-2 concurrent) -- Use quantized models (Q4, Q5) -- Monitor VRAM usage -- Automatic fallback to CPU for lightweight models - -### Risk 2: Qdrant Performance Degradation -**Impact:** Medium - Slower retrieval affects UX -**Probability:** Low - Qdrant is fast with proper indexing -**Mitigation:** -- Use HNSW indexing (default) -- Implement collection partitioning -- Add query filters to reduce search space -- Cache frequent queries - -### Risk 3: OpenAI API Incompatibility -**Impact:** High - Open WebUI won't work -**Probability:** Low - Spec is well-defined -**Mitigation:** -- Follow OpenAI API spec exactly -- Test thoroughly with Open WebUI -- Document unsupported features -- Keep Ollama direct as fallback - -### Risk 4: Complex Agent Workflows Timeout -**Impact:** Medium - Some tasks fail -**Probability:** Medium - Research tasks can be slow -**Mitigation:** -- Set reasonable timeouts (5 minutes) -- Implement streaming progress updates -- Break down complex tasks -- Return partial results on timeout - -### Risk 5: Memory Consolidation Overhead -**Impact:** Low - Slight performance impact -**Probability:** High - Consolidation is CPU intensive -**Mitigation:** -- Run consolidation async (background) -- Batch consolidation operations -- Use lightweight model for summaries -- Monitor consolidation performance - -## Conclusion - -This implementation plan provides a comprehensive roadmap for building a sophisticated AI orchestration layer that transforms the tower-of-joy infrastructure from a basic LLM chat interface into an intelligent, multi-agent system with proper memory, tool integration, and research capabilities. - -The phased approach ensures steady progress with testable milestones. The architecture is designed to integrate seamlessly with existing infrastructure (Ollama, Qdrant) while providing a foundation for future enhancements like mobile apps and home automation. - -**Current Status (Phase 4 Complete):** -- ✅ OpenAI-compatible API for Open WebUI -- ✅ Three-tier memory system with semantic recall -- ✅ Web search and research capabilities -- ✅ PydanticAI framework with reliable tool calling -- ✅ Production-grade performance and reliability - -**Next Steps:** -- Phase 5: Multi-agent patterns and RAG optimization -- Phase 6: Production hardening with monitoring -- Phase 7+: Advanced integrations (Nextcloud, ComfyUI, Home Assistant) - -This establishes the tower-of-joy project as a cutting-edge AI homelab with capabilities rivaling commercial solutions, all running on local hardware with full data sovereignty. - ---- - -**Plan Status:** ✅ Phase 4 Complete - PydanticAI Migration Successful -**Completed Phases:** -- Phase 1: Foundation (2025-11-13) -- Phase 2: Memory Systems (2025-11-23) -- Phase 3: Research Capabilities (2025-11-24) -- Phase 4: Framework Migration to PydanticAI (2025-11-26) - -**Next Steps:** Phase 5 - Multi-Agent Patterns & RAG Optimization (Future)