portainer-core/docs/phase1-implementation-guide.md

Phase 1 Implementation Guide: Foundation

Duration: Week 1 (5-7 days) Goal: Basic OpenAI-compatible API wrapper working with Open WebUI Status: Ready to implement

Overview

Phase 1 establishes the foundational API layer that makes Open WebUI think it's talking to OpenAI, while actually routing requests through our orchestrator to Ollama. This is the critical foundation that all future phases build upon.

End State: Open WebUI can connect to the orchestrator, send messages, and receive responses (both streaming and non-streaming) with zero regressions from the current Ollama setup.

Prerequisites

• Ollama running on port 11434
• Qdrant running on port 6333
• Open WebUI running on port 82
• Docker and Docker Compose available
• Python 3.12 environment
• Text editor or IDE ready

Project Structure

/home/jpmschweitzer/Projects/portainer-core/
└── services/
    └── ai-orchestrator/
        ├── Dockerfile
        ├── requirements.txt
        ├── .env.example
        ├── .dockerignore
        ├── README.md
        └── src/
            ├── __init__.py
            ├── main.py              # FastAPI entry point
            ├── config.py            # Configuration
            ├── api/
            │   ├── __init__.py
            │   ├── routes.py        # API routes
            │   └── schemas.py       # Pydantic models
            └── models/
                ├── __init__.py
                └── ollama_client.py # Ollama integration

Implementation Steps

Step 1: Create Project Structure (15 minutes)

Commands:

cd /home/jpmschweitzer/Projects/portainer-core

# Create directory structure
mkdir -p services/ai-orchestrator/src/api
mkdir -p services/ai-orchestrator/src/models

# Create __init__.py files
touch services/ai-orchestrator/src/__init__.py
touch services/ai-orchestrator/src/api/__init__.py
touch services/ai-orchestrator/src/models/__init__.py

Verification:

tree services/ai-orchestrator/src

Expected output:

services/ai-orchestrator/src
├── __init__.py
├── api
│   └── __init__.py
└── models
    └── __init__.py

---

Step 2: Create requirements.txt (5 minutes)

File: services/ai-orchestrator/requirements.txt

# FastAPI and server
fastapi==0.115.0
uvicorn[standard]==0.32.0
pydantic==2.10.4
pydantic-settings==2.7.0

# HTTP client
httpx==0.28.1

# Environment
python-dotenv==1.0.1

# Utilities
python-json-logger==2.0.7

Save the file, then verify:

cat services/ai-orchestrator/requirements.txt

---

Step 3: Create Configuration Module (10 minutes)

File: services/ai-orchestrator/src/config.py

"""
Configuration management for AI Orchestrator.
Loads settings from environment variables with sensible defaults.
"""

from pydantic_settings import BaseSettings, SettingsConfigDict
from typing import List


class Settings(BaseSettings):
    """Application settings loaded from environment variables."""

    # Application
    app_name: str = "AI Orchestrator"
    app_version: str = "1.0.0"
    debug: bool = False
    log_level: str = "INFO"

    # Server
    host: str = "0.0.0.0"
    port: int = 8084

    # Ollama
    ollama_base_url: str = "http://ollama:11434"
    ollama_timeout: int = 300  # 5 minutes

    # Model configuration
    default_model: str = "gemma:7b"
    lightweight_models: List[str] = ["gemma:2b", "gemma:7b"]
    heavy_models: List[str] = ["mistral:7b", "gemma2:9b"]
    code_models: List[str] = ["codestral:latest", "codegemma:latest"]

    # Model aliases (OpenAI → Local)
    model_aliases: dict = {
        "gpt-3.5-turbo": "gemma:7b",
        "gpt-4": "mistral:7b",
        "gpt-4-turbo": "mixtral:8x7b",
        "gpt-4-code": "codestral:latest",
    }

    # CORS
    cors_origins: List[str] = ["*"]

    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        case_sensitive=False
    )


# Global settings instance
settings = Settings()

Test it:

cd services/ai-orchestrator
python3 -c "from src.config import settings; print(f'Default model: {settings.default_model}')"

Expected: Default model: gemma:7b

---

Step 4: Create API Schemas (20 minutes)

File: services/ai-orchestrator/src/api/schemas.py

"""
OpenAI-compatible API schemas.
Pydantic models for request/response validation.
"""

from pydantic import BaseModel, Field
from typing import List, Optional, Dict, Any, Literal
from enum import Enum


# ============================================================================
# Request Schemas
# ============================================================================

class MessageRole(str, Enum):
    """Valid message roles."""
    SYSTEM = "system"
    USER = "user"
    ASSISTANT = "assistant"


class ChatMessage(BaseModel):
    """A single message in the conversation."""
    role: MessageRole
    content: str
    name: Optional[str] = None


class ChatCompletionRequest(BaseModel):
    """OpenAI-compatible chat completion request."""

    model: str = Field(..., description="Model to use")
    messages: List[ChatMessage] = Field(..., min_length=1)
    stream: bool = Field(default=False, description="Enable streaming")

    # Optional parameters
    temperature: Optional[float] = Field(default=0.7, ge=0, le=2)
    top_p: Optional[float] = Field(default=1.0, ge=0, le=1)
    max_tokens: Optional[int] = Field(default=None, ge=1)
    frequency_penalty: Optional[float] = Field(default=0.0, ge=-2, le=2)
    presence_penalty: Optional[float] = Field(default=0.0, ge=-2, le=2)
    stop: Optional[List[str]] = None

    class Config:
        json_schema_extra = {
            "example": {
                "model": "gpt-3.5-turbo",
                "messages": [
                    {"role": "user", "content": "Hello!"}
                ],
                "stream": False,
                "temperature": 0.7
            }
        }


# ============================================================================
# Response Schemas
# ============================================================================

class ChatMessageResponse(BaseModel):
    """Response message."""
    role: str = "assistant"
    content: str


class ChatCompletionChoice(BaseModel):
    """A single completion choice."""
    index: int = 0
    message: ChatMessageResponse
    finish_reason: str = "stop"


class UsageInfo(BaseModel):
    """Token usage information."""
    prompt_tokens: int = 0
    completion_tokens: int = 0
    total_tokens: int = 0


class ChatCompletionResponse(BaseModel):
    """OpenAI-compatible chat completion response (non-streaming)."""

    id: str
    object: str = "chat.completion"
    created: int
    model: str
    choices: List[ChatCompletionChoice]
    usage: UsageInfo


# ============================================================================
# Streaming Response Schemas
# ============================================================================

class DeltaMessage(BaseModel):
    """Delta message for streaming."""
    role: Optional[str] = None
    content: Optional[str] = None


class ChatCompletionStreamChoice(BaseModel):
    """Streaming choice."""
    index: int = 0
    delta: DeltaMessage
    finish_reason: Optional[str] = None


class ChatCompletionStreamResponse(BaseModel):
    """OpenAI-compatible streaming chunk."""

    id: str
    object: str = "chat.completion.chunk"
    created: int
    model: str
    choices: List[ChatCompletionStreamChoice]


# ============================================================================
# Other Endpoints
# ============================================================================

class ModelInfo(BaseModel):
    """Model information."""
    id: str
    object: str = "model"
    created: int = 0
    owned_by: str = "local"


class ModelsListResponse(BaseModel):
    """List of available models."""
    object: str = "list"
    data: List[ModelInfo]


class HealthCheckResponse(BaseModel):
    """Health check response."""
    status: str
    version: str
    ollama_connected: bool

Test it:

cd services/ai-orchestrator
python3 -c "from src.api.schemas import ChatCompletionRequest; print('Schemas loaded successfully')"

---

Step 5: Create Ollama Client (30 minutes)

File: services/ai-orchestrator/src/models/ollama_client.py

"""
Ollama client for model inference.
Handles both streaming and non-streaming requests.
"""

import httpx
import json
import logging
from typing import AsyncIterator, Dict, Any, Optional
from ..config import settings

logger = logging.getLogger(__name__)


class OllamaClient:
    """Client for interacting with Ollama API."""

    def __init__(self):
        self.base_url = settings.ollama_base_url
        self.timeout = settings.ollama_timeout
        self.client = httpx.AsyncClient(timeout=self.timeout)

    async def close(self):
        """Close the HTTP client."""
        await self.client.aclose()

    def resolve_model(self, model_name: str) -> str:
        """
        Resolve model alias to actual Ollama model.

        Args:
            model_name: Requested model name (e.g., "gpt-3.5-turbo")

        Returns:
            Actual Ollama model name (e.g., "gemma:7b")
        """
        resolved = settings.model_aliases.get(model_name, model_name)
        logger.info(f"Model resolution: {model_name} → {resolved}")
        return resolved

    async def generate_non_streaming(
        self,
        model: str,
        prompt: str,
        temperature: float = 0.7,
        max_tokens: Optional[int] = None
    ) -> Dict[str, Any]:
        """
        Generate non-streaming response from Ollama.

        Args:
            model: Model name
            prompt: User prompt
            temperature: Sampling temperature
            max_tokens: Maximum tokens to generate

        Returns:
            Dict with 'response' and 'tokens' keys
        """
        actual_model = self.resolve_model(model)

        payload = {
            "model": actual_model,
            "prompt": prompt,
            "stream": False,
            "options": {
                "temperature": temperature,
            }
        }

        if max_tokens:
            payload["options"]["num_predict"] = max_tokens

        logger.debug(f"Ollama request: {json.dumps(payload, indent=2)}")

        try:
            response = await self.client.post(
                f"{self.base_url}/api/generate",
                json=payload
            )
            response.raise_for_status()
            result = response.json()

            return {
                "response": result.get("response", ""),
                "tokens": {
                    "prompt": result.get("prompt_eval_count", 0),
                    "completion": result.get("eval_count", 0),
                    "total": result.get("prompt_eval_count", 0) + result.get("eval_count", 0)
                }
            }

        except httpx.HTTPError as e:
            logger.error(f"Ollama request failed: {e}")
            raise

    async def generate_streaming(
        self,
        model: str,
        prompt: str,
        temperature: float = 0.7,
        max_tokens: Optional[int] = None
    ) -> AsyncIterator[str]:
        """
        Generate streaming response from Ollama.

        Args:
            model: Model name
            prompt: User prompt
            temperature: Sampling temperature
            max_tokens: Maximum tokens to generate

        Yields:
            Token strings
        """
        actual_model = self.resolve_model(model)

        payload = {
            "model": actual_model,
            "prompt": prompt,
            "stream": True,
            "options": {
                "temperature": temperature,
            }
        }

        if max_tokens:
            payload["options"]["num_predict"] = max_tokens

        logger.debug(f"Ollama streaming request: {json.dumps(payload, indent=2)}")

        try:
            async with self.client.stream(
                "POST",
                f"{self.base_url}/api/generate",
                json=payload
            ) as response:
                response.raise_for_status()

                async for line in response.aiter_lines():
                    if not line:
                        continue

                    try:
                        chunk = json.loads(line)
                        if "response" in chunk:
                            token = chunk["response"]
                            if token:
                                yield token

                        # Check if done
                        if chunk.get("done", False):
                            break

                    except json.JSONDecodeError:
                        logger.warning(f"Failed to parse JSON: {line}")
                        continue

        except httpx.HTTPError as e:
            logger.error(f"Ollama streaming request failed: {e}")
            raise

    async def health_check(self) -> bool:
        """
        Check if Ollama is healthy.

        Returns:
            True if healthy, False otherwise
        """
        try:
            response = await self.client.get(
                f"{self.base_url}/api/tags",
                timeout=5.0
            )
            return response.status_code == 200
        except Exception as e:
            logger.error(f"Ollama health check failed: {e}")
            return False


# Global client instance
ollama_client = OllamaClient()

Test it:

cd services/ai-orchestrator
python3 -c "from src.models.ollama_client import OllamaClient; print('OllamaClient loaded successfully')"

---

Step 6: Create API Routes (45 minutes)

File: services/ai-orchestrator/src/api/routes.py

"""
API routes for OpenAI-compatible endpoints.
"""

import time
import json
import logging
from fastapi import APIRouter, HTTPException
from fastapi.responses import StreamingResponse, JSONResponse
from typing import AsyncIterator

from .schemas import (
    ChatCompletionRequest,
    ChatCompletionResponse,
    ChatCompletionChoice,
    ChatMessageResponse,
    UsageInfo,
    ChatCompletionStreamResponse,
    ChatCompletionStreamChoice,
    DeltaMessage,
    ModelsListResponse,
    ModelInfo,
    HealthCheckResponse
)
from ..models.ollama_client import ollama_client
from ..config import settings

logger = logging.getLogger(__name__)

router = APIRouter()


# ============================================================================
# Helper Functions
# ============================================================================

def build_prompt_from_messages(messages: list) -> str:
    """
    Convert message list to a simple prompt string.

    In Phase 1, we do simple concatenation.
    Phase 2 will add proper memory management.
    """
    prompt_parts = []

    for msg in messages:
        role = msg.role.value if hasattr(msg.role, 'value') else msg.role
        content = msg.content

        if role == "system":
            prompt_parts.append(f"System: {content}")
        elif role == "user":
            prompt_parts.append(f"User: {content}")
        elif role == "assistant":
            prompt_parts.append(f"Assistant: {content}")

    prompt_parts.append("Assistant:")
    return "\n\n".join(prompt_parts)


async def stream_chat_completion(
    request_id: str,
    model: str,
    prompt: str,
    temperature: float,
    max_tokens: int | None
) -> AsyncIterator[str]:
    """
    Stream chat completion in OpenAI SSE format.

    Yields:
        Server-Sent Events formatted strings
    """
    created = int(time.time())

    # First chunk with role
    first_chunk = ChatCompletionStreamResponse(
        id=request_id,
        created=created,
        model=model,
        choices=[
            ChatCompletionStreamChoice(
                index=0,
                delta=DeltaMessage(role="assistant"),
                finish_reason=None
            )
        ]
    )
    yield f"data: {first_chunk.model_dump_json()}\n\n"

    # Stream tokens
    try:
        async for token in ollama_client.generate_streaming(
            model=model,
            prompt=prompt,
            temperature=temperature,
            max_tokens=max_tokens
        ):
            chunk = ChatCompletionStreamResponse(
                id=request_id,
                created=created,
                model=model,
                choices=[
                    ChatCompletionStreamChoice(
                        index=0,
                        delta=DeltaMessage(content=token),
                        finish_reason=None
                    )
                ]
            )
            yield f"data: {chunk.model_dump_json()}\n\n"

    except Exception as e:
        logger.error(f"Streaming error: {e}")
        # Send error in OpenAI format
        error_chunk = {
            "error": {
                "message": str(e),
                "type": "server_error"
            }
        }
        yield f"data: {json.dumps(error_chunk)}\n\n"
        return

    # Final chunk
    final_chunk = ChatCompletionStreamResponse(
        id=request_id,
        created=created,
        model=model,
        choices=[
            ChatCompletionStreamChoice(
                index=0,
                delta=DeltaMessage(),
                finish_reason="stop"
            )
        ]
    )
    yield f"data: {final_chunk.model_dump_json()}\n\n"
    yield "data: [DONE]\n\n"


# ============================================================================
# Routes
# ============================================================================

@router.post("/v1/chat/completions")
async def chat_completions(request: ChatCompletionRequest):
    """
    OpenAI-compatible chat completions endpoint.
    Supports both streaming and non-streaming.
    """
    request_id = f"chatcmpl-{int(time.time() * 1000)}"

    logger.info(
        f"Chat request: id={request_id}, model={request.model}, "
        f"messages={len(request.messages)}, stream={request.stream}"
    )

    # Build prompt from messages
    prompt = build_prompt_from_messages(request.messages)

    # Streaming response
    if request.stream:
        return StreamingResponse(
            stream_chat_completion(
                request_id=request_id,
                model=request.model,
                prompt=prompt,
                temperature=request.temperature,
                max_tokens=request.max_tokens
            ),
            media_type="text/event-stream"
        )

    # Non-streaming response
    try:
        result = await ollama_client.generate_non_streaming(
            model=request.model,
            prompt=prompt,
            temperature=request.temperature,
            max_tokens=request.max_tokens
        )

        response = ChatCompletionResponse(
            id=request_id,
            created=int(time.time()),
            model=request.model,
            choices=[
                ChatCompletionChoice(
                    index=0,
                    message=ChatMessageResponse(
                        role="assistant",
                        content=result["response"]
                    ),
                    finish_reason="stop"
                )
            ],
            usage=UsageInfo(
                prompt_tokens=result["tokens"]["prompt"],
                completion_tokens=result["tokens"]["completion"],
                total_tokens=result["tokens"]["total"]
            )
        )

        logger.info(
            f"Chat response: id={request_id}, "
            f"tokens={result['tokens']['total']}"
        )

        return response

    except Exception as e:
        logger.error(f"Chat completion error: {e}")
        raise HTTPException(
            status_code=500,
            detail=f"Failed to generate completion: {str(e)}"
        )


@router.get("/v1/models")
async def list_models():
    """List available models."""

    # Map local models to OpenAI-style model IDs
    models = [
        ModelInfo(id=alias, owned_by="local")
        for alias in settings.model_aliases.keys()
    ]

    # Add actual local models
    for model_list in [
        settings.lightweight_models,
        settings.heavy_models,
        settings.code_models
    ]:
        for model in model_list:
            if model not in [m.id for m in models]:
                models.append(ModelInfo(id=model, owned_by="local"))

    return ModelsListResponse(data=models)


@router.get("/health")
async def health_check():
    """Health check endpoint."""

    ollama_healthy = await ollama_client.health_check()

    return HealthCheckResponse(
        status="healthy" if ollama_healthy else "degraded",
        version=settings.app_version,
        ollama_connected=ollama_healthy
    )


@router.get("/")
async def root():
    """Root endpoint."""
    return {
        "name": settings.app_name,
        "version": settings.app_version,
        "status": "running"
    }

---

Step 7: Create Main Application (20 minutes)

File: services/ai-orchestrator/src/main.py

"""
AI Orchestrator - Main FastAPI application.
OpenAI-compatible API for LangGraph agent orchestration.
"""

import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware

from .config import settings
from .api.routes import router
from .models.ollama_client import ollama_client

# Configure logging
logging.basicConfig(
    level=getattr(logging, settings.log_level),
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
logger = logging.getLogger(__name__)


@asynccontextmanager
async def lifespan(app: FastAPI):
    """Lifespan context manager for startup and shutdown."""
    # Startup
    logger.info(f"Starting {settings.app_name} v{settings.app_version}")
    logger.info(f"Ollama URL: {settings.ollama_base_url}")
    logger.info(f"Default model: {settings.default_model}")

    # Check Ollama connectivity
    ollama_healthy = await ollama_client.health_check()
    if ollama_healthy:
        logger.info("✓ Ollama connection successful")
    else:
        logger.warning("✗ Ollama connection failed - some features may not work")

    yield

    # Shutdown
    logger.info("Shutting down...")
    await ollama_client.close()


# Create FastAPI app
app = FastAPI(
    title=settings.app_name,
    version=settings.app_version,
    description="OpenAI-compatible API for multi-agent LLM orchestration",
    lifespan=lifespan
)

# Add CORS middleware
app.add_middleware(
    CORSMiddleware,
    allow_origins=settings.cors_origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Include routes
app.include_router(router)


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(
        "src.main:app",
        host=settings.host,
        port=settings.port,
        reload=settings.debug
    )

Test it:

cd services/ai-orchestrator
python3 -c "from src.main import app; print('FastAPI app loaded successfully')"

---

Step 8: Create Dockerfile (15 minutes)

File: services/ai-orchestrator/Dockerfile

FROM python:3.12-slim

# Prevent Python from writing pyc files and buffering
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONPATH=/app

WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements first (better layer caching)
COPY requirements.txt .

# Install Python dependencies
RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY ./src /app/src

# Create non-root user
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /app

USER appuser

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
    CMD curl -f http://localhost:8084/health || exit 1

# Expose port
EXPOSE 8084

# Run FastAPI with Uvicorn
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8084"]

---

Step 9: Create .dockerignore (5 minutes)

File: services/ai-orchestrator/.dockerignore

__pycache__
*.pyc
*.pyo
*.pyd
.Python
*.so
*.egg
*.egg-info
dist
build
.env
.venv
venv
.git
.gitignore
.pytest_cache
.coverage
htmlcov
.mypy_cache
.ruff_cache
*.log
.DS_Store
README.md
docs/
tests/

---

Step 10: Create Docker Compose Stack (15 minutes)

File: /home/jpmschweitzer/Projects/portainer-core/stacks/ai-orchestrator.yml

version: '3.8'

# AI Orchestrator - Phase 1: Foundation
# OpenAI-compatible API wrapper for Ollama
# Port: 8084

services:
  ai-orchestrator:
    build:
      context: /home/jpmschweitzer/Projects/portainer-core/services/ai-orchestrator
      dockerfile: Dockerfile
    container_name: ai-orchestrator
    restart: unless-stopped

    ports:
      - "8084:8084"

    environment:
      # Application
      - APP_NAME=AI Orchestrator
      - APP_VERSION=1.0.0-phase1
      - DEBUG=true
      - LOG_LEVEL=INFO

      # Server
      - HOST=0.0.0.0
      - PORT=8084

      # Ollama connection
      - OLLAMA_BASE_URL=http://ollama:11434
      - OLLAMA_TIMEOUT=300

      # Model configuration
      - DEFAULT_MODEL=gemma:7b
      - LIGHTWEIGHT_MODELS=gemma:2b,gemma:7b
      - HEAVY_MODELS=mistral:7b,gemma2:9b,mixtral:8x7b
      - CODE_MODELS=codestral:latest,codegemma:latest

      # Model aliases (OpenAI → Local)
      - MODEL_ALIAS_GPT35=gemma:7b
      - MODEL_ALIAS_GPT4=mistral:7b
      - MODEL_ALIAS_GPT4_TURBO=mixtral:8x7b
      - MODEL_ALIAS_GPT4_CODE=codestral:latest

      # CORS
      - CORS_ORIGINS=http://192.168.86.149:82,http://open-webui:8080

    networks:
      - ai-dataplane

    depends_on:
      - ollama

    labels:
      - "com.centurylinklabs.watchtower.enable=false"  # Disable auto-updates during development

    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8084/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

networks:
  ai-dataplane:
    external: true

---

Step 11: Create README (10 minutes)

File: services/ai-orchestrator/README.md

# AI Orchestrator

OpenAI-compatible API for multi-agent LLM orchestration with Ollama.

## Phase 1: Foundation

Basic API wrapper providing OpenAI-compatible endpoints for Open WebUI.

### Features

- ✅ OpenAI-compatible `/v1/chat/completions` endpoint
- ✅ Streaming and non-streaming responses
- ✅ Model aliasing (gpt-3.5-turbo → gemma:7b, etc.)
- ✅ Health checks
- ✅ Model listing

### Quick Start

```bash
# Build and deploy
cd /home/jpmschweitzer/Projects/portainer-core
docker-compose -f stacks/ai-orchestrator.yml up -d

# Check health
curl http://localhost:8084/health

# List models
curl http://localhost:8084/v1/models

# Test chat (non-streaming)
curl http://localhost:8084/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

Configuration

Environment variables (see docker-compose):

• OLLAMA_BASE_URL: Ollama API URL
• DEFAULT_MODEL: Default model for requests
• LOG_LEVEL: Logging level (DEBUG, INFO, WARNING, ERROR)

Development

# Install dependencies
pip install -r requirements.txt

# Run locally
python -m src.main

# Or with uvicorn
uvicorn src.main:app --reload --host 0.0.0.0 --port 8084

Next Steps

• Phase 2: Memory systems (3-tier architecture)
• Phase 3: Multi-agent workflows (LangGraph)
• Phase 4: Tool integration
• Phase 5: RAG and hybrid search
• Phase 6: Production hardening

---

## Testing Plan

### Test 1: Local Python Test (Before Docker)

```bash
cd /home/jpmschweitzer/Projects/portainer-core/services/ai-orchestrator

# Install dependencies in a venv (optional but recommended)
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Test imports
python3 -c "from src.main import app; print('✓ App imports successfully')"

# Run locally (requires Ollama accessible at localhost:11434)
# Temporarily change OLLAMA_BASE_URL to http://localhost:11434
OLLAMA_BASE_URL=http://localhost:11434 uvicorn src.main:app --host 0.0.0.0 --port 8084

Expected Output:

INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Starting AI Orchestrator v1.0.0-phase1
INFO:     Ollama URL: http://localhost:11434
INFO:     Default model: gemma:7b
INFO:     ✓ Ollama connection successful
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8084

Test in another terminal:

# Health check
curl http://localhost:8084/health

# Expected: {"status":"healthy","version":"1.0.0-phase1","ollama_connected":true}

# List models
curl http://localhost:8084/v1/models | jq

# Test chat
curl http://localhost:8084/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Say hello in one word"}],
    "stream": false
  }' | jq

---

Test 2: Docker Build Test

cd /home/jpmschweitzer/Projects/portainer-core/services/ai-orchestrator

# Build image
docker build -t ai-orchestrator:phase1 .

# Expected: Successfully built image

# Verify image
docker images | grep ai-orchestrator

# Expected: ai-orchestrator   phase1   [image-id]   [size]

---

Test 3: Docker Compose Deployment

cd /home/jpmschweitzer/Projects/portainer-core

# Deploy stack
docker-compose -f stacks/ai-orchestrator.yml up -d

# Check container
docker ps | grep ai-orchestrator

# Check logs
docker logs ai-orchestrator

# Expected logs:
# INFO:     Starting AI Orchestrator v1.0.0-phase1
# INFO:     Ollama URL: http://ollama:11434
# INFO:     ✓ Ollama connection successful

---

Test 4: API Functionality Tests

Test health endpoint:

curl http://192.168.86.149:8084/health

Expected:

{
  "status": "healthy",
  "version": "1.0.0-phase1",
  "ollama_connected": true
}

Test models endpoint:

curl http://192.168.86.149:8084/v1/models | jq '.data[].id'

Expected:

"gpt-3.5-turbo"
"gpt-4"
"gpt-4-turbo"
"gpt-4-code"
"gemma:2b"
"gemma:7b"
...

Test non-streaming chat:

curl http://192.168.86.149:8084/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "user", "content": "Count from 1 to 5"}
    ],
    "stream": false,
    "temperature": 0.7
  }' | jq

Expected:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1699564800,
  "model": "gpt-3.5-turbo",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "1\n2\n3\n4\n5"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 12,
    "total_tokens": 22
  }
}

Test streaming chat:

curl http://192.168.86.149:8084/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "user", "content": "Say hello"}
    ],
    "stream": true
  }'

Expected (streaming output):

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":...,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":...,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":...,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":...,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

---

Test 5: Open WebUI Integration

Update Open WebUI configuration:

# Edit open-webui stack
nano /home/jpmschweitzer/Projects/portainer-core/stacks/open-webui.yml

# Add this environment variable:
environment:
  - OPENAI_API_BASE=http://ai-orchestrator:8084/v1
  - ENABLE_OPENAI_API=true
  - OLLAMA_BASE_URL=http://ollama:11434  # Keep as fallback

# Restart Open WebUI
docker-compose -f stacks/open-webui.yml down
docker-compose -f stacks/open-webui.yml up -d

Test in Open WebUI:

1. Open http://192.168.86.149:82
2. Go to Settings → Connections
3. Verify you can see both endpoints
4. Select "OpenAI API"
5. Choose model "gpt-3.5-turbo" (which maps to gemma:7b)
6. Start a conversation: "Hello, testing Phase 1"
7. Verify response arrives
8. Try streaming: Should see tokens appear one by one

---

Success Criteria Checklist

Code Quality

• All Python files have proper imports
• No syntax errors
• Type hints where appropriate
• Docstrings for functions
• Logging configured

Functionality

• Health endpoint returns 200
• Models endpoint lists available models
• Non-streaming chat works
• Streaming chat works
• Model aliases resolve correctly
• Error handling works (try invalid model name)

Docker

• Image builds successfully
• Container starts without errors
• Container passes health check
• Can connect to Ollama from container
• Logs are readable

Integration

• Open WebUI can connect
• Can send messages through orchestrator
• Responses appear in Open WebUI
• Streaming works in UI
• No regressions from direct Ollama connection

---

Troubleshooting

Issue: "Ollama connection failed"

Symptoms: Health check shows ollama_connected: false

Solutions:

1. Check Ollama is running: docker ps | grep ollama
2. Check network connectivity: docker exec ai-orchestrator curl http://ollama:11434/api/tags
3. Verify both containers on ai-dataplane network
4. Check Ollama logs: docker logs ollama

Issue: "Model not found"

Symptoms: Error when trying to use a model

Solutions:

1. List available models in Ollama: docker exec ollama ollama list
2. Pull missing model: docker exec ollama ollama pull gemma:7b
3. Check model aliases in config.py
4. Verify model name spelling

Issue: "Streaming not working"

Symptoms: No streaming output, or all text arrives at once

Solutions:

1. Check Content-Type header: Should be text/event-stream
2. Verify stream=true in request
3. Check browser/client supports SSE
4. Test with curl first (easier to debug)

Issue: "Open WebUI can't connect"

Symptoms: Open WebUI shows connection error

Solutions:

1. Verify orchestrator is on ai-dataplane network
2. Check Open WebUI environment variable: OPENAI_API_BASE=http://ai-orchestrator:8084/v1
3. Test from Open WebUI container: docker exec open-webui curl http://ai-orchestrator:8084/health
4. Check Open WebUI logs: docker logs open-webui

---

Next Steps After Phase 1

Once Phase 1 is complete and all tests pass:

1. Document any issues encountered - Add to troubleshooting
2. Commit to git (if using version control)
3. Update STATUS.md - Mark Phase 1 complete
4. Review with stakeholders - Demo the working system
5. Plan Phase 2 kickoff - Memory systems implementation

---

Time Estimates

Step	Task	Time
1	Create structure	15 min
2	requirements.txt	5 min
3	config.py	10 min
4	schemas.py	20 min
5	ollama_client.py	30 min
6	routes.py	45 min
7	main.py	20 min
8	Dockerfile	15 min
9	.dockerignore	5 min
10	docker-compose	15 min
11	README.md	10 min
Coding Total		~3 hours
Test 1	Local test	30 min
Test 2	Docker build	15 min
Test 3	Deploy	15 min
Test 4	API tests	30 min
Test 5	Open WebUI	30 min
Testing Total		~2 hours
Grand Total		~5 hours

Add buffer time for debugging: 1-2 hours

Total realistic time: 6-7 hours (one work day)

---

Ready to start implementation? I can help you with any step!