portainer-core/plans/active/ai-orchestrator-plan.md

AI Orchestrator Implementation Plan

Project: tower-of-joy AI Stack Enhancement Created: 2025-11-13 Status: Phase 4 Complete ✅ - ADK Migration Successful Updated: 2025-11-26 Framework: Google ADK 1.3.0 with LiteLLM 1.80.5

Executive Summary

This document outlines the implementation of a sophisticated AI orchestration layer using Google ADK and FastAPI that provides an advanced agent system for the tower-of-joy homelab. The architecture provides:

✅ MIGRATION COMPLETE (2025-11-26): Successfully migrated from LangChain/LangGraph to Google ADK to achieve reliable tool calling with Ollama local models. See MIGRATION_PLAN_LANGCHAIN_TO_ADK.md for details.

• Advanced Memory Systems: Three-tier memory with Qdrant for long-term semantic recall
• Multi-Agent Workflows: Intelligent routing to lightweight, heavy, and specialist models
• Extensive Tool Integration: Web search, file operations, calendar, home automation, image generation
• Production-Ready API: OpenAI-compatible endpoints for seamless Open WebUI integration
• Superior Performance: Proper context management, caching, and model selection

Current vs Target Architecture

Current Architecture (v0.6.0)

┌──────────────┐
│  Open WebUI  │
│  (Port 82)   │
└──────┬───────┘
       │
       │ Direct connection
       │
┌──────▼───────┐      ┌──────────────┐
│   Ollama     │      │   Qdrant     │
│  (Port 11434)│      │  (Port 6333) │
└──────────────┘      └──────────────┘
       │
       │ GPU inference
       │
┌──────▼───────┐
│  RTX 2080 Ti │
│   (11GB)     │
└──────────────┘

Limitations:

• Open WebUI's memory integration not working well
• No intelligent model routing
• Limited tool calling capabilities
• Single-model processing (no multi-agent coordination)
• Difficult to customize RAG behavior

Target Architecture

┌────────────────────────────────────────────────────────────────┐
│                      User Interface Layer                       │
│                   Open WebUI (Port 82)                          │
└───────────────────────────┬────────────────────────────────────┘
                            │
                            │ /v1/chat/completions (OpenAI-compatible)
                            │
┌───────────────────────────▼────────────────────────────────────┐
│                  AI Orchestrator (Port 8084)                    │
│  ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓  │
│  ┃          FastAPI + LangGraph Orchestration Layer        ┃  │
│  ┃  • OpenAI-compatible API wrapper                        ┃  │
│  ┃  • Request routing & agent coordination                 ┃  │
│  ┃  • Memory management (3-tier system)                    ┃  │
│  ┗━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛  │
│                           │                                     │
│    ┌──────────────────────┼──────────────────────┐             │
│    │                      │                      │             │
│  ┌─▼──────────┐   ┌───────▼───────┐   ┌─────────▼──────┐     │
│  │ Chat Agent │   │ Research Agent│   │  Tool Agent    │     │
│  │ (General)  │   │ (Deep search) │   │  (Actions)     │     │
│  └─────┬──────┘   └───────┬───────┘   └─────────┬──────┘     │
└────────┼──────────────────┼─────────────────────┼─────────────┘
         │                  │                     │
         │                  │                     │
┌────────▼──────────────────▼─────────────────────▼─────────────┐
│              Model Inference Layer (Ollama)                     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐        │
│  │ Lightweight  │  │    Heavy     │  │  Specialist  │        │
│  │ gemma:2b     │  │ mistral:7b   │  │ codestral    │        │
│  │ gemma:7b     │  │ gemma2:9b    │  │ codegemma    │        │
│  │              │  │ gemma2:27b   │  │ mixtral:8x7b │        │
│  │ ~2-4GB VRAM  │  │ ~6-8GB VRAM  │  │ ~8-10GB VRAM │        │
│  └──────────────┘  └──────────────┘  └──────────────┘        │
└───────────────────────────────────────────────────────────────┘
         │
         │ GPU acceleration
         │
┌────────▼───────────────────────────────────────────────────────┐
│                     NVIDIA RTX 2080 Ti (11GB VRAM)              │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────┐    ┌───────────────────────────────┐
│   Memory & Storage      │    │   External Tools & APIs       │
│   (Qdrant Port 6333)    │    │                               │
│                         │    │  • Core API (web scraper)     │
│  Tier 1: Working Memory │    │  • Nextcloud API (files)      │
│  Tier 2: Summaries      │    │  • Calendar (CalDAV)          │
│  Tier 3: Vector Store   │    │  • ComfyUI (image gen)        │
│  • Conversations        │    │  • Home Assistant (IoT)       │
│  • Documents            │    │  • Web Search (DuckDuckGo)    │
│  • User facts           │    │  • Future integrations        │
└─────────────────────────┘    └───────────────────────────────┘

Advantages:

• ✅ Intelligent multi-model routing (right model for the task)
• ✅ Proper conversation memory with semantic recall
• ✅ Multi-agent coordination for complex tasks
• ✅ Extensive tool calling (web search, files, calendar, automation)
• ✅ Research agents for deep information gathering
• ✅ Image generation via ComfyUI integration
• ✅ Gradual migration path (run parallel with current setup)
• ✅ Foundation for custom mobile apps later

Technology Stack

Core Framework (Current - Post-Migration)

• Google ADK 1.3.0 - Agent Development Kit for stateful agents
• LiteLLM 1.80.5 - Unified LLM interface for Ollama compatibility
• FastAPI 0.115.0 - REST API framework
• Uvicorn ≥0.34.0 - ASGI server
• Pydantic ≥2.11.1 - Request/response validation

AI & Memory

• google-genai 1.17.0 - Google Generative AI SDK
• google-cloud-aiplatform 1.95.1 - AI Platform integration
• qdrant-client ~1.11.0 - Vector database client
• Ollama - Local LLM inference (via LiteLLM)

Utilities

• httpx ≥0.28.0 - Async HTTP client for external APIs
• python-dotenv ~1.0.0 - Environment configuration
• duckduckgo-search ~4.1.0 - Web search integration
• beautifulsoup4 ~4.12.0 - Web scraping
• trafilatura ~1.12.0 - Content extraction

Container

• Python 3.12 - Runtime (already upgraded)
• Docker - Containerization
• Network: ai-dataplane (shared with Ollama, Qdrant, Open WebUI)

Three-Tier Memory Architecture

Tier 1: Working Memory (In-Memory)

Purpose: Immediate context for ongoing conversation

Implementation: ConversationBufferMemory

• Stores last 10 conversation turns in RAM
• Fast access (< 1ms)
• Automatic pruning when limit reached
• Lost on container restart (ephemeral)

Storage: 0MB persistent, ~5KB RAM

Tier 2: Short-Term Memory (SQLite)

Purpose: Recent conversation summaries

Implementation: ConversationSummaryMemory

• Summarized conversation history (hours to days)
• Stored in SQLite database
• Medium access speed (~10ms)
• Persists across restarts

Storage: ~/docker-data/ai-orchestrator/data/memory.db (~500KB per 100 conversations)

Tier 3: Long-Term Memory (Qdrant)

Purpose: Semantic search across entire conversation history

Implementation: VectorStoreRetrieverMemory with Qdrant

• All conversations embedded and stored as vectors
• Semantic similarity search for relevant context
• Unlimited history retention
• Fast semantic search (< 50ms)

Storage: Qdrant collection conversation_memory (~1KB per turn, 10MB for 10k turns)

Memory Consolidation Strategy

# Consolidation triggers
CONSOLIDATION_RULES = {
    "message_count": 10,        # Every 10 messages → Summarize to Tier 2
    "token_limit": 2000,        # When context > 2000 tokens → Compress
    "conversation_end": True,   # End of conversation → Embed to Tier 3
    "explicit_save": True,      # User: "remember this" → Force save
}

Multi-Agent Workflow System

Router Agent (Lightweight Model)

Model: gemma:2b or gemma:7b Purpose: Analyze incoming requests and route to appropriate agent/model

Decision Criteria:

• Task complexity (token estimation, keyword analysis)
• Domain specialization (code, math, general, creative)
• Tool requirements (web search, file access, image generation)
• Response quality needs (fast vs accurate)

Chat Agent (General Purpose)

Model: mistral:7b (default) or gemma:7b (simple queries) Purpose: Handle general conversation, Q&A, casual interactions

Capabilities:

• Normal chat interactions
• Simple questions and answers
• Memory recall from Qdrant
• Basic tool calling (web search, file access)

Research Agent (Deep Analysis)

Model: mixtral:8x7b or mistral:7b Purpose: Complex research tasks requiring web search and synthesis

Workflow:

1. Query expansion (generate related search terms)
2. Web search (DuckDuckGo, multiple queries)
3. Content scraping (via Core API)
4. Analysis (extract key information)
5. Synthesis (generate comprehensive report)

Tools:

• Web search
• Web scraping (Core API)
• Document retrieval (Qdrant)

Code Agent (Specialist)

Model: codestral:latest or codegemma:latest Purpose: Programming tasks, debugging, code generation

Capabilities:

• Code generation (multiple languages)
• Debugging and optimization
• Code explanation
• API integration examples

Tool Agent (Action Executor)

Model: mistral:7b Purpose: Execute actions using external tools and APIs

Available Tools:

• Web Search (DuckDuckGo)
• Web Scraping (Core API)
• File Operations (Nextcloud API)
• Calendar Management (CalDAV via Nextcloud)
• Image Generation (ComfyUI/Stable Diffusion)
• Home Automation (Home Assistant - future)
• Task Management (future custom system)

OpenAI-Compatible API Design

Endpoint: POST /v1/chat/completions

Request Schema:

{
  "model": "gpt-3.5-turbo",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Hello!"}
  ],
  "stream": false,
  "temperature": 0.7,
  "max_tokens": 2048
}

Model Aliasing:

MODEL_ALIASES = {
    "gpt-3.5-turbo": "gemma:7b",           # Fast, lightweight
    "gpt-4": "mistral:7b",                 # High quality
    "gpt-4-turbo": "mixtral:8x7b",         # Very capable
    "gpt-4-code": "codestral:latest",      # Code specialist
    "gpt-4-32k": "gemma2:27b",             # Longer context
}

Response Schema (Non-Streaming):

{
  "id": "chatcmpl-1234567890",
  "object": "chat.completion",
  "created": 1699564800,
  "model": "gpt-3.5-turbo",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Hello! How can I help you?"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 8,
    "total_tokens": 18
  }
}

Response Schema (Streaming):

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1699564800,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Additional Endpoints

• GET /v1/models - List available models
• GET /health - Health check
• GET /metrics - Prometheus metrics
• POST /v1/embeddings - Generate embeddings (future)

Tool Integration Plan

Phase 1 Tools (Core Functionality)

1. Web Search Tool

Integration: DuckDuckGo API (no API key required) Purpose: Find current information on the web

class WebSearchTool(BaseTool):
    name = "web_search"
    description = "Search the web for current information"

    def _run(self, query: str, num_results: int = 5) -> str:
        """Execute web search via DuckDuckGo."""

2. Web Scraping Tool

Integration: Core API (already deployed) Purpose: Extract content from web pages

class WebScrapeTool(BaseTool):
    name = "scrape_webpage"
    description = "Extract clean text content from a URL"

    def _run(self, url: str) -> str:
        """Call Core API scraper endpoint."""
        response = httpx.post(
            "http://core-api:8083/scrape",
            json={"url": url}
        )

3. Document Search Tool

Integration: Qdrant documents collection Purpose: Search uploaded documents and previous conversations

class DocumentSearchTool(BaseTool):
    name = "search_documents"
    description = "Search through uploaded documents and conversation history"

    def _run(self, query: str) -> str:
        """Semantic search in Qdrant."""

Phase 2 Tools (Productivity)

4. Nextcloud File Tool

Integration: Nextcloud WebDAV API Purpose: Search and access files in Nextcloud

class NextcloudFileTool(BaseTool):
    name = "search_files"
    description = "Search for files in Nextcloud"

5. Calendar Tool

Integration: CalDAV via Nextcloud Purpose: Check calendar, add events

class CalendarTool(BaseTool):
    name = "check_calendar"
    description = "Check calendar for events or add new events"

Phase 3 Tools (Advanced)

6. Image Generation Tool

Integration: ComfyUI API (when deployed) Purpose: Generate images from text descriptions

class ImageGenerationTool(BaseTool):
    name = "generate_image"
    description = "Generate images using Stable Diffusion"

7. Home Automation Tool

Integration: Home Assistant API (when deployed) Purpose: Control smart home devices

class HomeAssistantTool(BaseTool):
    name = "control_home"
    description = "Control smart home devices"

8. Task Management Tool

Integration: Custom task system (future) Purpose: Create, read, update tasks and reminders

class TaskManagementTool(BaseTool):
    name = "manage_tasks"
    description = "Create and manage tasks and reminders"

Directory Structure

/home/jpmschweitzer/Projects/portainer-core/
├── services/
│   └── ai-orchestrator/
│       ├── Dockerfile
│       ├── requirements.txt
│       ├── .env.example
│       ├── README.md
│       └── src/
│           ├── __init__.py
│           ├── main.py              # FastAPI app entry point
│           ├── config.py            # Configuration management
│           │
│           ├── api/                 # API layer
│           │   ├── __init__.py
│           │   ├── routes.py        # Route definitions
│           │   ├── schemas.py       # Pydantic models
│           │   └── middleware.py    # Auth, CORS, logging
│           │
│           ├── agents/              # LangGraph agents
│           │   ├── __init__.py
│           │   ├── router.py        # Router agent
│           │   ├── chat.py          # Chat agent
│           │   ├── research.py      # Research agent
│           │   ├── code.py          # Code agent
│           │   └── tool_executor.py # Tool agent
│           │
│           ├── memory/              # Memory systems
│           │   ├── __init__.py
│           │   ├── working.py       # Tier 1 (in-memory)
│           │   ├── summary.py       # Tier 2 (SQLite)
│           │   ├── vector.py        # Tier 3 (Qdrant)
│           │   └── consolidation.py # Memory consolidation
│           │
│           ├── models/              # Model management
│           │   ├── __init__.py
│           │   ├── router.py        # Model selection logic
│           │   ├── aliases.py       # Model name mapping
│           │   └── manager.py       # Model lifecycle
│           │
│           ├── tools/               # LangChain tools
│           │   ├── __init__.py
│           │   ├── web_search.py    # DuckDuckGo search
│           │   ├── web_scrape.py    # Core API scraper
│           │   ├── document.py      # Qdrant document search
│           │   ├── nextcloud.py     # File operations
│           │   ├── calendar.py      # Calendar management
│           │   ├── image_gen.py     # Image generation
│           │   └── home_assistant.py # Home automation
│           │
│           ├── rag/                 # RAG system
│           │   ├── __init__.py
│           │   ├── retriever.py     # Hybrid retrieval
│           │   ├── embeddings.py    # Embedding generation
│           │   └── reranker.py      # Re-ranking
│           │
│           └── utils/               # Utilities
│               ├── __init__.py
│               ├── logging.py       # Structured logging
│               ├── metrics.py       # Prometheus metrics
│               └── helpers.py       # Common utilities
│
└── stacks/
    └── ai-orchestrator.yml          # Docker Compose stack

/home/jpmschweitzer/docker-data/
└── ai-orchestrator/
    ├── data/
    │   ├── memory.db                # SQLite for summaries
    │   └── checkpoints/             # LangGraph checkpoints
    ├── logs/
    │   └── app.log                  # Application logs
    └── cache/                       # Response cache

Implementation Phases

Phase 1: Foundation (Week 1) ✅ COMPLETED 2025-11-13

Goal: Basic OpenAI-compatible API wrapper that works with Open WebUI

Status: ✅ All tasks completed. Implementation added to Core API service.

Tasks: ✅ ALL COMPLETE

1. ✅ Create service directory structure (extended Core API instead)
2. ✅ Implement FastAPI app with /v1/chat/completions endpoint
3. ✅ Add OpenAI request/response schemas (Pydantic models)
4. ✅ Connect to Ollama for model inference
5. ✅ Implement basic streaming support (SSE format)
6. ✅ Add model aliasing (gpt-3.5-turbo → gemma:7b)
7. ✅ Create Dockerfile and requirements.txt (reused Core API container)
8. ✅ Create Docker Compose stack definition (updated core-api.yml)
9. ✅ Deploy to ai-dataplane network
10. ✅ Test with Open WebUI

Deliverables: ✅ ALL DELIVERED

• ✅ Working /v1/chat/completions endpoint (src/api/v1/chat.py)
• ✅ Both streaming and non-streaming responses
• ✅ Model name mapping (src/config.py model_aliases)
• ✅ Docker container deployed (core-api on port 8083)
• ✅ /v1/models endpoint (src/api/v1/models.py)
• ✅ OllamaClient with connection management (src/models/ollama_client.py)

Success Criteria: ✅ ALL MET

• ✅ OpenAI-compatible API responding correctly
• ✅ Streaming works properly (Server-Sent Events format)
• ✅ Non-streaming responses working
• ✅ Model aliasing functional (tested gpt-3.5-turbo → gemma:7b)
• ✅ Health check passing, Ollama connectivity verified

Implementation Notes:

• Implemented within existing Core API service rather than separate container
• Hot-reload development mode active for rapid iteration
• Ready for Open WebUI integration (endpoint: http://core-api:8083/v1)

Phase 2: Memory Systems (Week 2)

Goal: Persistent conversation memory with three-tier architecture

Tasks:

1. Implement Tier 1: ConversationBufferMemory (in-memory)
2. Implement Tier 2: ConversationSummaryMemory (SQLite)
3. Integrate Tier 3: VectorStoreRetrieverMemory (Qdrant)
4. Create Qdrant collections (conversation_memory, documents, user_facts)
5. Implement memory consolidation service
6. Add conversation history API endpoints
7. Build memory recall in conversation flow
8. Test memory persistence across container restarts
9. Add memory metrics (Prometheus)

Deliverables:

• Three-tier memory system
• Persistent conversation storage
• Memory consolidation pipeline
• Conversation recall functionality
• Memory metrics dashboard

Success Criteria:

• Conversations persist across restarts
• Memory recall provides relevant context
• Semantic search returns appropriate results
• No memory leaks or unbounded growth

Phase 3: Research Capabilities ✅ COMPLETE (2025-11-24)

Goal: Web search and research workflows

Status: ✅ COMPLETE - All success criteria met (LangChain implementation)

Completed Tasks:

1. ✅ Integrated DuckDuckGo web search API
2. ✅ Implemented automatic content scraping from search results
3. ✅ Added web_search tool (search + scrape in one call)
4. ✅ Added web_scrape tool (targeted URL extraction)
5. ✅ Enhanced system prompts for research query detection
6. ✅ Added enhanced progress indicators (🔍, 📄 icons)
7. ✅ Comprehensive testing (6/6 tests passed, 100% success rate)
8. ✅ Tool invocation logging for debugging
9. ✅ System prompt A/B testing (6 variants, v1_verbose winner)
10. ✅ Model validation (mistral:7b confirmed best for tools)

Deliverables: ✅ ALL DELIVERED

• ✅ Web search with DuckDuckGo integration
• ✅ Automatic content extraction from results
• ✅ Research query detection in system prompts
• ✅ 9 total tools (7 infrastructure + 2 research)
• ✅ Enhanced user experience with visual feedback
• ✅ Comprehensive test suite with 100% success
• ✅ Tool logging infrastructure

Success Criteria: ✅ ALL MET

• ✅ Research detection accuracy: 100% (target: >80%)
• ✅ Average response time: 5.3s (target: <10s)
• ✅ Source citation rate: 100% (target: >90%)
• ✅ Tool calling reliability: 100% on complex queries
• ✅ No regression in existing functionality

Implementation Details:

• Approach: Extended unified agent (Option A from Phase 3 plan)
• Framework: LangChain/LangGraph ReAct agent
• Model: mistral:7b (validated via extensive testing)
• Dependencies: duckduckgo-search~=4.1.0
• Performance: 4-7s for research, 0.3s for simple chat

Testing & Optimization:

• System Prompt A/B Testing: 6 variants tested across 30 scenarios
• Winner: v1_verbose (87/100 score, 0.92s avg response)
• Identified placeholder text issue (models mimic examples)
• Tool logging added for production debugging

Deferred to Future Phases:

• Multi-agent routing patterns (Phase 4/5)
• Code specialist agent with codestral (Phase 4/5)
• Model switching based on complexity (Phase 4/5)
• Supervisor pattern for agent coordination (Phase 5)

See Also:

• Phase 3 Completion Document
• System Prompt Test Results
• Lightweight Model Testing

Phase 4: Framework Migration ✅ COMPLETE (2025-11-26)

Goal: Migrate from LangChain/LangGraph to Google ADK for improved reliability

Status: ✅ COMPLETE - All success criteria met

Completed Tasks:

1. ✅ Updated requirements.txt (removed langchain*, added google-adk, litellm)
2. ✅ Rewrote orchestrator.py to use ADK Agent with LiteLLM
3. ✅ Converted all 9 tools to ADK async generator format
4. ✅ Updated streaming.py for ADK event format
5. ✅ Upgraded model from mistral:7b to gemma3:12b
6. ✅ Created v7_adk_best_practice system prompt variant
7. ✅ Enhanced health checks for agent monitoring
8. ✅ Production testing and validation
9. ✅ Documentation updates

Deliverables: ✅ ALL DELIVERED

• ✅ Google ADK 1.3.0 integration with LiteLLM 1.80.5
• ✅ 9 tools migrated to ADK format (7 infrastructure + 2 research)
• ✅ Model upgrade to gemma3:12b (~8GB VRAM)
• ✅ ADK-optimized system prompts
• ✅ Improved streaming consistency
• ✅ Enhanced agent health monitoring
• ✅ Complete documentation

Success Criteria: ✅ ALL MET

• ✅ Tool calling works reliably (no more empty tool_calls arrays)
• ✅ Gemma models now supported (was failing with LangChain)
• ✅ Streaming output consistent and clean
• ✅ No regressions in memory system
• ✅ API endpoints unchanged (backward compatible)
• ✅ Performance within targets

Implementation Details:

• Framework: Google ADK with UnifiedAgent pattern
• Model Bridge: LiteLLM for Ollama compatibility
• Model: gemma3:12b (upgraded from mistral:7b)
• Prompt: v7_adk_best_practice
• Tools: All 9 tools as ADK async generators
• Migration Time: ~6 hours (as estimated)

Performance (Post-Migration):

• Simple queries: ~0.3-1s
• Tool-using queries: ~2-5s (improved from LangChain)
• Research queries: ~4-7s (maintained)
• VRAM usage: ~8GB with gemma3:12b

Why This Migration:

• ❌ Problem: LangGraph's create_react_agent failed to trigger tools with Ollama
• ❌ Problem: Gemma models returned status 400 with LangChain
• ❌ Problem: Inconsistent streaming behavior
• ✅ Solution: ADK has proven Ollama integration via LiteLLM
• ✅ Result: Reliable tool calling across all models

See Also:

• Migration Plan
• ADK/Ollama Research
• Changelog Entry

Phase 5: Tool Integration & RAG Optimization (Future)

Goal: Enhanced tool calling and advanced RAG capabilities

Tasks:

1. Create LangChain tool interface base class
2. Implement Web Search Tool (DuckDuckGo)
3. Implement Web Scrape Tool (Core API integration)
4. Implement Document Search Tool (Qdrant)
5. Test tool calling in agent workflows
6. Add tool usage metrics
7. Implement tool error handling and retries
8. Add tool result caching
9. Create tool documentation
10. Test agent tool usage in real scenarios

Deliverables:

• 3 working tools (search, scrape, document)
• Tool calling in agents
• Error handling and retries
• Tool metrics
• Usage documentation

Success Criteria:

• Agents can successfully call tools
• Web search returns relevant results
• Web scraping extracts clean content
• Document search finds relevant info
• Tools handle errors gracefully

Phase 5: RAG & Advanced Memory (Week 5)

Goal: Document retrieval and hybrid search

Tasks:

1. Implement hybrid retrieval (dense + sparse)
2. Create document embedding pipeline
3. Build RAG chain with Qdrant
4. Add re-ranking for better results
5. Integrate RAG with conversation flow
6. Add document upload endpoint
7. Implement document chunking strategy
8. Test RAG with various document types
9. Optimize retrieval performance
10. Add RAG metrics

Deliverables:

• Hybrid search system (semantic + keyword)
• Document embedding pipeline
• RAG-enhanced responses
• Re-ranking optimization
• Document upload API

Success Criteria:

• Documents can be uploaded and indexed
• Semantic search returns relevant chunks
• Hybrid search improves accuracy
• RAG responses use document context
• Performance meets targets (< 100ms retrieval)

Phase 6: Production Hardening (Week 6)

Goal: Observability, error handling, optimization

Tasks:

1. Add structured logging (structlog)
2. Implement comprehensive error handling
3. Add retry logic for external calls
4. Implement Prometheus metrics
5. Create health check endpoints
6. Add request/response caching
7. Optimize model selection logic
8. Performance testing and optimization
9. Load testing (concurrent requests)
10. Documentation and deployment guide

Deliverables:

• Production-ready service
• Monitoring and metrics
• Error handling
• Performance benchmarks
• Load test results
• Complete documentation

Success Criteria:

• Structured logs for debugging
• All errors handled gracefully
• Metrics exported to Prometheus
• Health checks pass
• Response times < 2s (p95)
• Can handle 10+ concurrent requests
• Documentation complete

Docker Configuration

Dockerfile

FROM python:3.12-slim

# Prevent Python from writing pyc files and buffering
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONPATH=/app

WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements first (better caching)
COPY requirements.txt .
RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY ./src /app/src

# Create non-root user
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /app
USER appuser

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
    CMD curl -f http://localhost:8084/health || exit 1

# Expose port
EXPOSE 8084

# Run FastAPI with Uvicorn
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8084", "--workers", "1"]

Docker Compose Stack (stacks/ai-orchestrator.yml)

version: '3.8'

# AI Orchestrator - LangGraph/LangChain service with OpenAI-compatible API
# Purpose: Intelligent agent workflows with multi-model routing and advanced memory
# Port: 8084 (HTTP API)
# Network: ai-dataplane (shared with Ollama, Qdrant, Open WebUI)
# Dependencies: Ollama (models), Qdrant (memory), Core API (web scraping)

services:
  ai-orchestrator:
    build:
      context: /home/jpmschweitzer/Projects/portainer-core/services/ai-orchestrator
      dockerfile: Dockerfile
    container_name: ai-orchestrator
    restart: unless-stopped

    ports:
      - "8084:8084"

    environment:
      # Application
      - APP_NAME=AI Orchestrator
      - APP_VERSION=1.0.0
      - DEBUG=false
      - LOG_LEVEL=INFO
      - ENVIRONMENT=production

      # Server
      - HOST=0.0.0.0
      - PORT=8084
      - WORKERS=1

      # Model endpoints
      - OLLAMA_BASE_URL=http://ollama:11434
      - QDRANT_URL=http://qdrant:6333
      - CORE_API_URL=http://core-api:8083

      # Model configuration
      - DEFAULT_MODEL=gemma:7b
      - LIGHTWEIGHT_MODELS=gemma:2b,gemma:7b
      - HEAVY_MODELS=mistral:7b,gemma2:9b,mixtral:8x7b
      - CODE_MODELS=codestral:latest,codegemma:latest
      - MATH_MODELS=mistral:7b

      # Model aliases (OpenAI → Local)
      - MODEL_ALIAS_GPT35=gemma:7b
      - MODEL_ALIAS_GPT4=mistral:7b
      - MODEL_ALIAS_GPT4_TURBO=mixtral:8x7b
      - MODEL_ALIAS_GPT4_CODE=codestral:latest

      # Memory configuration
      - MEMORY_COLLECTION=conversation_memory
      - MAX_WORKING_MEMORY=10
      - CONSOLIDATION_INTERVAL=10
      - ENABLE_MEMORY_CONSOLIDATION=true

      # RAG configuration
      - RAG_ENABLED=true
      - EMBEDDING_MODEL=nomic-embed-text
      - RETRIEVAL_K=5
      - HYBRID_SEARCH=true
      - RERANK_ENABLED=true

      # Agent configuration
      - MAX_ITERATIONS=10
      - AGENT_TIMEOUT=300
      - ENABLE_RESEARCH_AGENT=true
      - ENABLE_CODE_AGENT=true
      - ENABLE_TOOL_AGENT=true

      # Tool configuration
      - ENABLE_WEB_SEARCH=true
      - ENABLE_WEB_SCRAPE=true
      - ENABLE_DOCUMENT_SEARCH=true
      - ENABLE_NEXTCLOUD=false
      - ENABLE_CALENDAR=false
      - ENABLE_IMAGE_GEN=false
      - ENABLE_HOME_ASSISTANT=false

      # Performance
      - ENABLE_CACHING=true
      - CACHE_TTL=3600
      - MAX_CONCURRENT_REQUESTS=10

      # Security
      - CORS_ORIGINS=http://192.168.86.149:82,http://open-webui:8080
      - API_KEY_REQUIRED=false
      # - API_KEY=your-secret-key-here

      # Monitoring
      - ENABLE_METRICS=true
      - METRICS_PORT=9090

    volumes:
      # Persistent data (SQLite, checkpoints)
      - /home/jpmschweitzer/docker-data/ai-orchestrator/data:/app/data

      # Logs
      - /home/jpmschweitzer/docker-data/ai-orchestrator/logs:/app/logs

      # Cache
      - /home/jpmschweitzer/docker-data/ai-orchestrator/cache:/app/cache

      # Optional: Mount source for development (hot reload)
      # - /home/jpmschweitzer/Projects/portainer-core/services/ai-orchestrator/src:/app/src

    networks:
      - ai-dataplane

    depends_on:
      - ollama
      - qdrant

    labels:
      - "com.centurylinklabs.watchtower.enable=true"

    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8084/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

networks:
  ai-dataplane:
    external: true

# Deployment Notes:
# 1. Ensure Ollama and Qdrant are running first
# 2. Create data directories: mkdir -p ~/docker-data/ai-orchestrator/{data,logs,cache}
# 3. Deploy in Portainer: Stacks → Add Stack → Upload this file
# 4. Verify health: curl http://localhost:8084/health
# 5. Test API: curl http://localhost:8084/v1/models
# 6. Configure Open WebUI to use this endpoint

Migration Strategy

Parallel Deployment Approach

The orchestrator will be deployed alongside the existing Ollama connection, allowing gradual migration with easy rollback.

Phase 1: Deploy Orchestrator (Week 1)

• Deploy ai-orchestrator container
• Keep Open WebUI pointing to Ollama directly
• Test orchestrator independently using curl/httpx

Phase 2: Dual Configuration (Week 2)

• Configure Open WebUI with both endpoints:
  • Primary: Ollama (http://ollama:11434) - Existing
  • Secondary: AI Orchestrator (http://ai-orchestrator:8084/v1) - New

Users can choose which endpoint to use in Open WebUI settings.

Phase 3: Gradual Migration (Weeks 3-4)

• Test orchestrator extensively
• Gather user feedback
• Fix issues as they arise
• Demonstrate superior capabilities (memory, tools, research)

Phase 4: Primary Switch (Week 5)

• Make orchestrator the default endpoint
• Keep Ollama direct as fallback option
• Monitor for any issues

Phase 5: Full Migration (Week 6)

• If stable, make orchestrator the only endpoint
• Document the change
• Keep Ollama direct as admin-only option

Rollback Plan

If issues arise at any point:

1. Switch Open WebUI back to Ollama direct connection
2. Debug orchestrator issues offline
3. Fix and re-test before re-enabling
4. No downtime for users

Open WebUI Configuration

Current Configuration (stacks/open-webui.yml):

environment:
  - OLLAMA_BASE_URL=http://ollama:11434

Dual Configuration (Migration Phase):

environment:
  - OLLAMA_BASE_URL=http://ollama:11434  # Fallback
  - OPENAI_API_BASE=http://ai-orchestrator:8084/v1  # New
  - ENABLE_OPENAI_API=true

Final Configuration (After Migration):

environment:
  - OLLAMA_BASE_URL=http://ai-orchestrator:8084/v1  # Primary
  # - OLLAMA_FALLBACK_URL=http://ollama:11434  # Emergency fallback

Performance Targets

Response Time Targets

Scenario	Target (p50)	Target (p95)	Target (p99)
Simple chat (lightweight model)	< 500ms	< 1s	< 2s
Complex chat (heavy model)	< 1s	< 2s	< 4s
Research task (multi-step)	< 5s	< 10s	< 15s
Tool calling (web search)	< 2s	< 4s	< 6s
RAG retrieval	< 100ms	< 200ms	< 500ms

Throughput Targets

Metric	Target
Concurrent requests	10+
Requests per minute	60+
GPU utilization	60-80%
Memory usage (orchestrator)	< 1GB

Quality Targets

Metric	Target
Model routing accuracy	> 90%
Memory recall relevance	> 85%
Tool calling success rate	> 95%
API compatibility	100% (OpenAI spec)

Monitoring & Observability

Metrics to Track (Prometheus)

Request Metrics:

• ai_orchestrator_requests_total{model, status} - Total requests
• ai_orchestrator_request_duration_seconds{model} - Request latency histogram
• ai_orchestrator_active_requests - Currently processing requests

Agent Metrics:

• ai_orchestrator_agent_invocations_total{agent_type} - Agent usage
• ai_orchestrator_agent_duration_seconds{agent_type} - Agent execution time
• ai_orchestrator_model_routing_total{from_model, to_model} - Routing decisions

Memory Metrics:

• ai_orchestrator_memory_consolidations_total - Memory consolidations
• ai_orchestrator_memory_retrieval_duration_seconds - Retrieval time
• ai_orchestrator_memory_size_bytes{tier} - Memory size per tier

Tool Metrics:

• ai_orchestrator_tool_calls_total{tool_name, status} - Tool usage
• ai_orchestrator_tool_duration_seconds{tool_name} - Tool execution time
• ai_orchestrator_tool_errors_total{tool_name} - Tool failures

Logging Strategy

Structured Logging with structlog:

logger.info(
    "chat_request_received",
    request_id=request_id,
    model=request.model,
    message_count=len(request.messages),
    stream=request.stream,
    user_id=user_id
)

Log Levels:

• DEBUG: Detailed agent workflows, tool calls, memory operations
• INFO: Request/response, agent routing decisions
• WARNING: Fallbacks, retries, degraded performance
• ERROR: Failures, exceptions, unrecoverable errors

Health Checks

Endpoint: GET /health

Checks:

• API status
• Ollama connectivity
• Qdrant connectivity
• Core API connectivity
• Memory system health
• Disk space

Response:

{
  "status": "healthy",
  "timestamp": 1699564800,
  "checks": {
    "api": "healthy",
    "ollama": "healthy",
    "qdrant": "healthy",
    "core_api": "healthy",
    "memory": "healthy",
    "disk": "healthy"
  },
  "version": "1.0.0"
}

Security Considerations

Authentication (Phase 6)

Optional API key authentication:

@app.post("/v1/chat/completions")
async def chat_completions(
    request: ChatCompletionRequest,
    api_key: str = Depends(verify_api_key)
):
    # ... process request

Rate Limiting (Phase 6)

Prevent abuse:

@limiter.limit("60/minute")  # 60 requests per minute per IP
async def chat_completions(...):
    # ... process request

Input Validation

Pydantic models validate all inputs:

class ChatCompletionRequest(BaseModel):
    model: constr(min_length=1, max_length=100)
    messages: List[ChatMessage]
    max_tokens: Optional[conint(ge=1, le=4096)] = None

CORS Configuration

Restrict origins:

app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://192.168.86.149:82"],  # Open WebUI
    allow_credentials=True,
    allow_methods=["POST", "GET"],
    allow_headers=["*"]
)

Testing Strategy

Unit Tests

• Test individual components (router, memory, tools)
• Mock external dependencies (Ollama, Qdrant)
• Use pytest and pytest-asyncio

Integration Tests

• Test complete API endpoints
• Real connections to Ollama/Qdrant
• Test streaming and non-streaming responses

Load Tests

• Test concurrent request handling
• Measure response times under load
• Identify bottlenecks

End-to-End Tests

• Test via Open WebUI
• Test complex multi-agent workflows
• Test tool calling and RAG

Documentation Plan

User Documentation

• API documentation (OpenAPI/Swagger)
• Model selection guide
• Memory system explanation
• Tool usage examples

Developer Documentation

• Architecture overview
• Code structure
• Adding new agents
• Adding new tools
• Configuration guide

Operations Documentation

• Deployment guide
• Monitoring setup
• Troubleshooting guide
• Performance tuning

Success Criteria

Phase 1 Success

• ✅ Open WebUI can connect and chat
• ✅ Streaming works correctly
• ✅ Model aliases function
• ✅ No errors in logs

Phase 2 Success

• ✅ Memory persists across restarts
• ✅ Semantic recall works
• ✅ Consolidation triggers properly
• ✅ No memory leaks

Phase 3 Success

• ✅ Agent routing works correctly
• ✅ Multi-step workflows complete
• ✅ Specialist agents activate appropriately
• ✅ State management functions

Phase 4 Success

• ✅ Tools callable from agents
• ✅ Web search returns results
• ✅ Web scraping extracts content
• ✅ Error handling works

Phase 5 Success

• ✅ Documents indexed and searchable
• ✅ Hybrid search improves results
• ✅ RAG provides relevant context
• ✅ Performance targets met

Phase 6 Success

• ✅ Metrics exported to Prometheus
• ✅ Health checks pass
• ✅ Load tests successful
• ✅ Documentation complete
• ✅ Production deployment successful

Future Enhancements (Post-Launch)

Phase 7+: Advanced Features

1. Nextcloud Integration - File search, calendar management
2. ComfyUI Integration - Image generation capabilities
3. Home Assistant Integration - Smart home control
4. Task Management System - Custom task/todo system
5. Mobile Apps - Native iOS/Android apps
6. Home Screen Widgets - Quick actions and status
7. Voice Interface - Voice command processing
8. Proactive Notifications - Intelligent reminders
9. Multi-User Support - Per-user memory and preferences
10. Fine-Tuned Models - Custom models for specific tasks

Risk Assessment & Mitigation

Risk 1: GPU VRAM Exhaustion

Impact: High - Service fails if VRAM exceeded Probability: Medium - Can happen with concurrent heavy model loads Mitigation:

• Implement model queue (max 1-2 concurrent)
• Use quantized models (Q4, Q5)
• Monitor VRAM usage
• Automatic fallback to CPU for lightweight models

Risk 2: Qdrant Performance Degradation

Impact: Medium - Slower retrieval affects UX Probability: Low - Qdrant is fast with proper indexing Mitigation:

• Use HNSW indexing (default)
• Implement collection partitioning
• Add query filters to reduce search space
• Cache frequent queries

Risk 3: OpenAI API Incompatibility

Impact: High - Open WebUI won't work Probability: Low - Spec is well-defined Mitigation:

• Follow OpenAI API spec exactly
• Test thoroughly with Open WebUI
• Document unsupported features
• Keep Ollama direct as fallback

Risk 4: Complex Agent Workflows Timeout

Impact: Medium - Some tasks fail Probability: Medium - Research tasks can be slow Mitigation:

• Set reasonable timeouts (5 minutes)
• Implement streaming progress updates
• Break down complex tasks
• Return partial results on timeout

Risk 5: Memory Consolidation Overhead

Impact: Low - Slight performance impact Probability: High - Consolidation is CPU intensive Mitigation:

• Run consolidation async (background)
• Batch consolidation operations
• Use lightweight model for summaries
• Monitor consolidation performance

Cost-Benefit Analysis

Development Cost

• Time: 6 weeks (1 developer)
• Infrastructure: $0 (using existing hardware)
• Opportunity cost: Medium (could work on other features)

Benefits

• Superior Memory: Proper conversation context and recall
• Multi-Agent Intelligence: Right model for each task
• Tool Integration: Web search, file access, automation
• Research Capabilities: Deep information gathering
• Future-Proof: Foundation for mobile apps and custom UI
• Better UX: Faster, more accurate, more capable

ROI

• Short-term: Improved AI interactions immediately
• Medium-term: Platform for advanced features
• Long-term: Foundation for custom AI applications

Conclusion

This implementation plan provides a comprehensive roadmap for building a sophisticated AI orchestration layer that transforms the tower-of-joy infrastructure from a basic LLM chat interface into an intelligent, multi-agent system with proper memory, tool integration, and research capabilities.

The phased approach ensures steady progress with testable milestones, while the parallel deployment strategy minimizes risk and allows for easy rollback if needed. The architecture is designed to integrate seamlessly with existing infrastructure (Ollama, Qdrant, Core API) while providing a foundation for future enhancements like mobile apps and home automation.

By the end of Week 6, the system will provide:

• ✅ OpenAI-compatible API for Open WebUI
• ✅ Three-tier memory system with semantic recall
• ✅ Multi-agent workflows with intelligent routing
• ✅ Tool integration (web search, scraping, documents)
• ✅ RAG with hybrid search
• ✅ Production-grade monitoring and observability

This establishes the tower-of-joy project as a cutting-edge AI homelab with capabilities rivaling commercial solutions, all running on local hardware with full data sovereignty.

---

Plan Status: ✅ Phase 4 Complete - ADK Migration Successful Completed Phases:

• Phase 1: Foundation (2025-11-13)
• Phase 2: Memory Systems (2025-11-23)
• Phase 3: Research Capabilities (2025-11-24)
• Phase 4: Framework Migration to ADK (2025-11-26)

Next Steps: Phase 5 - Multi-Agent Patterns & RAG Optimization (Future)