jpmschweitzer/portainer-core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(core-ai): comprehensive cleanup - PydanticAI only architecture

Browse Source 
Remove all obsolete agent implementations and framework references.
Keep only PydanticAI (primary) and SimpleLiteLLM (fallback).

This cleanup eliminates confusion between multiple frameworks that were
tried during development (LangChain, LangGraph, ADK, OllamaNative) and
establishes PydanticAI as the single agent framework going forward.

BREAKING CHANGES:
- Removed OllamaNativeAgent - use PydanticAgent instead
- Removed /test/ollama-tools diagnostic endpoint
- Default /v1/chat/completions now uses PydanticAgent

Files Deleted (32 total):
- Obsolete agents: ollama_native_agent.py
- Diagnostic files: ARCHITECTURE.md, DIAGNOSTIC_RESULTS.md, PHASE*.md
- Legacy tools: src/tools.py
- Test files: test_ai_flow_quality.py, test_02/03 (diagnostic layers)
- Documentation: ADK_Ollama_Research.md, agent-flow-diagrams.md
- Session docs: 3 files with LangChain/LangGraph implementations
- Plans: 5 completed plans about obsolete frameworks
- Migration docs: MIGRATION_PLAN_LANGCHAIN_TO_ADK.md

Files Modified (8 total):
- main.py: Refactored to PydanticAI only (305 lines vs 457 before)
- agents/__init__.py: Removed OllamaNativeAgent exports
- README.md: Complete rewrite for PydanticAI architecture
- prompts.py: Updated for PydanticAI (infrastructure tool guidance)
- STATUS.md: Updated to v0.11.0-pydantic-ai
- CHANGELOG.md: Added v0.11.0 entry documenting cleanup
- plans/active/*.md: Updated to reference PydanticAI

Current Architecture:
- Framework: PydanticAI with native Ollama SDK
- Agents: PydanticAgent (primary) + SimpleLiteLLMAgent (fallback)
- Model: mistral-nemo:latest
- Tools: 6 core + 28+ OpenAPI-discovered
- Memory: 3-tier system with Qdrant
- VRAM: ~4-6GB

Lines Removed: ~3000+ lines of obsolete code

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2025-12-03 14:24:52 +01:00
co-authored by Claude
parent 96492cb1ed
commit 66f6e54fc3
32 changed files with 1078 additions and 10376 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ADK_Ollama_Research.md
-84
View File
@@ -1,84 +0,0 @@
# Research on ADK, LiteLLM, and Ollama Integration in `core-api`
## 1. Introduction
This document provides a detailed analysis of the AI chat implementation within the `core-api` service, focusing on the integration of Google's Agent Development Kit (ADK), LiteLLM, and Ollama. The primary goal is to understand the current architecture, identify probable causes for production failures, and propose actionable improvements to enhance stability, maintainability, and performance.
## 2. Current Implementation Analysis
The `core-api` service employs a sophisticated but complex dual-path architecture for handling chat completions.
### 2.1. Dual-Path Architecture
Two distinct endpoints process chat requests:
1. **Agent-Based Path (`/api/v1/ai/chat/completions`):** Managed by `src/controllers/ai_controller.py`, this is the primary, advanced endpoint. It leverages an agent built with the Google ADK for complex logic, including tool usage. If the agent is unavailable or fails, this endpoint critically falls back to the direct path.
2. **Direct Ollama Path (`/api/v1/chat/completions`):** Defined in `src/api/v1/chat.py`, this endpoint provides a simpler, OpenAI-compatible interface that interacts directly with Ollama, bypassing the agent.
This dual-path system, especially the silent fallback in the main controller, creates ambiguity and can mask critical failures in the agent stack.
### 2.2. ADK and LiteLLM Integration
The core of the agent is in `src/agent/orchestrator.py`.
- It uses `google-adk` to define the agent's structure and logic (`UnifiedAgent`).
- It uses `litellm` as a compatibility layer to connect the ADK to the Ollama backend. The agent is instantiated on a per-request basis, making it stateless from the ADK's perspective.
- **Crucially, the connection to Ollama is configured via the `OLLAMA_API_BASE` environment variable.**
### 2.3. Configuration Management
Application settings are centralized in `src/config.py` and loaded from `.env` files. However, a critical inconsistency exists:
- The **direct Ollama client** (`src/models/ollama_client.py`) correctly uses the `ollama_base_url` setting from the `Settings` object.
- The **ADK/LiteLLM agent** (`src/agent/orchestrator.py`) ignores this and relies exclusively on the `OLLAMA_API_BASE` environment variable.
This discrepancy is a primary source of configuration fragility.
### 2.4. Production Environment
The `Dockerfile` defines the production container. It installs dependencies from `requirements.txt` (including `google-adk` and `litellm`) but **does not set the `OLLAMA_API_BASE` environment variable.** This means the agent defaults to LiteLLM's hardcoded `http://localhost:11434`, which may not be correct in all deployment scenarios.
The Docker `HEALTHCHECK` only tests the direct Ollama client via `/health`, meaning the service can report as healthy even if the entire agent stack is non-functional.
## 3. Potential Causes of Production Errors
The investigation points to several likely causes for the reported failures.
1. **Configuration Mismatch (Most Likely Cause):** The agent is likely failing because the `OLLAMA_API_BASE` environment variable is not set or is set incorrectly in the production environment. Because the direct client uses a different configuration variable (`ollama_base_url`), the fallback mechanism works, and the API returns a successful response, completely hiding the agent's failure. Developers may be unaware that the agent is not being used.
2. **Silent Agent Failure:** The fallback logic in `ai_controller.py` prevents any errors from the agent from propagating. While this ensures availability, it makes debugging impossible and hides the fact that advanced features (tool use, complex reasoning) are not executing.
3. **Incomplete Health Check:** The current health check provides a false sense of security. The service can be "healthy" while the core agent functionality is broken.
## 4. Suggested Improvements and Optimizations
To address these issues, the following improvements are recommended:
1. **Unify Configuration:**
- **Action:** Refactor `src/agent/orchestrator.py` to source the Ollama URL from the central `Settings` object in `src/config.py`. Remove the dependency on the `OLLAMA_API_BASE` environment variable.
- **Benefit:** Creates a single, unambiguous source of truth for the Ollama URL, simplifying configuration and reducing errors.
2. **Eliminate Redundant Endpoint:**
- **Action:** Deprecate and remove the `/api/v1/chat/completions` endpoint in `src/api/v1/chat.py`. The `ai_controller` should be the sole entry point for all chat-related requests.
- **Benefit:** Simplifies the architecture, removes code duplication, and eliminates confusion about which endpoint to use.
3. **Improve Health Checks:**
- **Action:** Implement a dedicated agent health check endpoint (e.g., `/health/agent`) that specifically invokes the agent and verifies its connection to Ollama via LiteLLM.
- **Benefit:** Provides a true signal of the agent's status, enabling reliable automated monitoring and faster failure detection.
4. **Introduce an Explicit Failure Mode:**
- **Action:** Add a configuration flag (e.g., `AGENT_FALLBACK_ENABLED`) that, when disabled in development/testing environments, causes agent failures to return a `500` error instead of silently falling back.
- **Benefit:** Makes debugging the agent significantly easier.
5. **Explore Stateful ADK Sessions:**
- **Action:** Investigate using the ADK's built-in session management (`session_service`). This would involve creating sessions that persist across multiple requests.
- **Benefit:** Could improve performance by reducing agent initialization overhead and would enable more sophisticated, multi-turn conversational memory within the agent's context.
## 5. Architectural Review and Validity
The current architecture is powerful and ambitious. The use of the Google ADK provides a solid foundation for building advanced, tool-using agents, and the Qdrant-based memory system is robust.
However, its validity is severely undermined by its fragility and opacity. The configuration mismatch and silent fallback mechanism make the system difficult to debug and unreliable in a production setting. The dual-path entry points add unnecessary complexity.
The architecture is fundamentally sound but requires the recommended refactoring to become robust, maintainable, and production-ready. By unifying configuration, improving observability, and simplifying the request flow, the `core-api` service can reliably deliver on the promise of its advanced agent capabilities.
docs/architecture/agent-flow-diagrams.md
-759
View File
@@ -1,759 +0,0 @@
# Agent Architecture Flow Diagrams
**Date**: 2025-11-23
**System**: Core API Unified Agent with LangGraph
This document shows the data flow through the agent system for various scenarios, including which models are used and how components interact.
---
## System Components Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ Open WebUI │
│ (or any OpenAI client) │
└────────────────────────┬────────────────────────────────────────┘
│ POST /v1/chat/completions
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Core API (FastAPI) │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ AI Controller (ai_controller.py) │ │
│ │ • Routes all requests to unified agent │ │
│ │ • Converts OpenAI format ↔ agent format │ │
│ └─────────┬────────────────────────────────────────┬───────┘ │
│ │ │ │
│ │ │ │
└────────────┼─────────────────────────────────────────────────────┘
│
▼
┌──────────────────────┐
│ Unified Agent │
│ (orchestrator.py) │
│ • LangGraph ReAct │
│ • mistral:7b │
│ • Tool calling │
│ • Decides: tools │
│ or direct answer │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Agent Tools │
│ (tools.py) │
│ • Infrastructure │
│ • Web scraping │
│ • Documentation │
└──────────────────────┘
```
---
## Scenario 1: Simple Knowledge Prompt (No Tools Needed)
**User**: _"What is Docker?"_
```
┌──────────┐
│ User │ "What is Docker?"
└────┬─────┘
│ POST /v1/chat/completions
│
▼
┌────────────────────────────────────────────┐
│ Core API - AI Controller │
│ │
│ 1. Parse request │
│ 2. Routes to unified agent │
│ 3. Extract message & history │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Unified Agent (orchestrator.py) │
│ │
│ Model: mistral:7b (tool-calling capable) │
│ │
│ System Prompt: │
│ "You are a homelab assistant..." │
│ │
│ Available Tools: │
│ - list_services │
│ - web_search │
│ - read_documentation │
│ - ... [7 tools total] │
└────┬───────────────────────────────────────┘
│
│ Agent reasoning:
│ "This is general knowledge,
│ no tools needed"
│
▼
┌────────────────────────────────────────────┐
│ LangGraph ReAct Loop │
│ │
│ [Thought] Analyzing query... │
│ [Decision] Direct answer, no tools │
│ [Action] Generate response │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Ollama (mistral:7b) │
│ │
│ Generates: "Docker is a platform for │
│ containerizing applications..." │
└────┬───────────────────────────────────────┘
│
│ [💭 Analyzing...] (thinking)
│ "Docker is a platform..." (content)
│
▼
┌────────────────────────────────────────────┐
│ Stream to SSE Format │
│ (streaming.py) │
│ │
│ Converts to OpenAI SSE chunks: │
│ data: {"choices":[{"delta":{"content":""}}]}│
└────┬───────────────────────────────────────┘
│
▼
┌──────────┐
│ User │ Sees: [💭 Analyzing...] → response
└──────────┘
```
**Models Used**:
- `mistral:7b` (agent reasoning + response generation)
**Data Flow**:
1. Request → AI Controller
2. AI Controller → Unified Agent
3. Agent → mistral:7b (direct query, no tools)
4. mistral:7b → Response text
5. Agent → SSE formatter → User
---
## Scenario 2: Web Search Required
**User**: _"What's the weather in San Francisco?"_
```
┌──────────┐
│ User │ "What's the weather in SF?"
└────┬─────┘
│
▼
┌────────────────────────────────────────────┐
│ AI Controller │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Thought] Need real-time weather data │
│ [Decision] Use web_search tool │
│ [Action] Call web_search( │
│ url="https://wttr.in/san-francisco" │
│ ) │
└────┬───────────────────────────────────────┘
│
│ Tool call
│
▼
┌────────────────────────────────────────────┐
│ Tool: web_search (tools.py) │
│ │
│ 1. Fetch URL via httpx │
│ 2. Extract content (trafilatura) │
│ 3. Return text content │
└────┬───────────────────────────────────────┘
│
│ Tool result: "Current: 62°F, Cloudy..."
│
▼
┌────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Observation] Got weather data │
│ [Thought] Format for user │
│ [Action] Generate final response │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Ollama (mistral:7b) │
│ │
│ Generates: "The weather in San Francisco │
│ is currently 62°F and cloudy..." │
└────┬───────────────────────────────────────┘
│
│ SSE stream:
│ [💭 Analyzing...] → [🔧 Searching web...] → [✓ Found data] → Response
│
▼
┌──────────┐
│ User │
└──────────┘
```
**Models Used**:
- `mistral:7b` (agent reasoning, tool selection, response synthesis)
**Data Flow**:
1. User → AI Controller → Agent
2. Agent analyzes → Decides to use `web_search`
3. Tool executes → Fetches web content
4. Tool result → Back to agent
5. Agent synthesizes → Final response
6. Stream to user with status indicators
**Components Involved**:
- AI Controller (routing)
- Unified Agent (orchestration)
- mistral:7b (reasoning at each step)
- web_search tool (httpx + trafilatura)
- SSE formatter (status indicators)
---
## Scenario 3: Code Generation from Swagger Docs
**User**: _"Write Python code to list all containers using the Core API"_
```
┌──────────┐
│ User │ "Write code to list containers"
└────┬─────┘
│
▼
┌────────────────────────────────────────────┐
│ AI Controller │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Thought] Need API docs to write accurate code │
│ [Decision] Use read_documentation tool │
│ [Action] read_documentation("swagger") │
└────┬───────────────────────────────────────────────────────┘
│
│ Tool call
│
▼
┌────────────────────────────────────────────────────────────┐
│ Tool: read_documentation (tools.py) │
│ │
│ 1. Reads /app/docs/openapi.json │
│ 2. Searches for container-related endpoints │
│ 3. Returns relevant API specs │
└────┬───────────────────────────────────────────────────────┘
│
│ Returns: GET /infrastructure/containers endpoint spec
│
▼
┌────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Observation] Found API endpoint details │
│ [Thought] Need to generate Python code │
│ [Decision] Could use code model for better quality │
│ │
│ ⚠️ Current: Uses mistral:7b for code generation │
│ 🔮 Future: Could route to codestral:latest │
└────┬───────────────────────────────────────────────────────┘
│
│ Generate code using API spec
│
▼
┌────────────────────────────────────────────────────────────┐
│ Ollama (mistral:7b) │
│ │
│ Synthesizes code based on: │
│ - API documentation │
│ - User request │
│ - Python best practices │
│ │
│ Output: │
│ ```python │
│ import httpx │
│ │
│ async def list_containers(): │
│ async with httpx.AsyncClient() as client: │
│ response = await client.get( │
│ "http://api.schweitz.net/infrastructure/..." │
│ ) │
│ return response.json() │
│ ``` │
└────┬───────────────────────────────────────────────────────┘
│
│ SSE stream:
│ [💭 Analyzing...] → [🔧 Reading docs...] → [✓ Found API] → Code output
│
▼
┌──────────┐
│ User │
└──────────┘
```
**Models Used**:
- `mistral:7b` (agent reasoning + code generation)
- **Future enhancement**: Could route to `codestral:latest` for code generation
**Data Flow**:
1. User → Agent
2. Agent → read_documentation tool
3. Tool → Reads OpenAPI spec from disk
4. Spec → Back to agent
5. Agent + spec → mistral:7b for code synthesis
6. Code → Stream to user
**Potential Optimization**:
```
┌────────────────────────────────────────────┐
│ Future: Model Routing │
│ │
│ Agent detects code generation request │
│ ↓ │
│ Routes to codestral:latest │
│ (instead of mistral:7b) │
│ ↓ │
│ Better code quality │
└────────────────────────────────────────────┘
```
---
## Scenario 4: Infrastructure Query
**User**: _"List all NPM proxy hosts and their domains"_
```
┌──────────┐
│ User │ "List NPM proxies and domains"
└────┬─────┘
│
▼
┌────────────────────────────────────────────┐
│ AI Controller │
└────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Thought] User wants NPM proxy configuration │
│ [Decision] Use list_domains tool │
│ [Action] list_domains() │
└────┬─────────────────────────────────────────────────────────┘
│
│ Tool call
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Tool: list_domains (tools.py) │
│ │
│ 1. Calls get_npm_client() │
│ 2. Makes request to NPM API: │
│ GET http://npm:81/api/nginx/proxy-hosts │
│ 3. Parses response │
│ 4. Extracts domain names & forwards │
└────┬─────────────────────────────────────────────────────────┘
│
│ Tool result:
│ [
│ {"domain": "home.schweitz.net", "forward": "organizr:80"},
│ {"domain": "api.schweitz.net", "forward": "core-api:8083"},
│ {"domain": "media.schweitz.net", "forward": "jellyfin:8096"},
│ ...
│ ]
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Observation] Got NPM proxy list │
│ [Thought] Format nicely for user │
│ [Action] Generate formatted response │
└────┬─────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Ollama (mistral:7b) │
│ │
│ Synthesizes response: │
│ │
│ "Here are your NPM proxy hosts: │
│ │
│ 1. home.schweitz.net → organizr:80 │
│ 2. api.schweitz.net → core-api:8083 │
│ 3. media.schweitz.net → jellyfin:8096 │
│ ..." │
└────┬─────────────────────────────────────────────────────────┘
│
│ SSE stream:
│ [💭 Analyzing...] → [🔧 Querying NPM...] → [✓ Found 12 proxies] → Response
│
▼
┌──────────┐
│ User │
└──────────┘
Data Path Detail:
═══════════════════
User Request
↓
AI Controller
↓
Unified Agent (mistral:7b)
↓
list_domains tool
↓
NPM Client (npm_client.py)
↓
HTTP Request → NPM Container (nginx-proxy-manager:81)
↓
NPM API Response (JSON)
↓
Parsed data → Tool
↓
Tool result → Agent
↓
mistral:7b synthesizes
↓
Formatted response
↓
SSE Stream → User
```
**Models Used**:
- `mistral:7b` (all reasoning + synthesis)
**Components in Data Path**:
1. **AI Controller** - Request routing
2. **Unified Agent** - Orchestration & reasoning (mistral:7b)
3. **list_domains Tool** - Business logic wrapper
4. **NPM Client** - HTTP client to NPM API
5. **NPM Container** - Actual nginx proxy manager
6. **SSE Formatter** - Stream status indicators
**External Systems**:
- Nginx Proxy Manager API (port 81)
---
## Scenario 5: Multi-Tool Complex Query
**User**: _"Which services are unhealthy and need to be restarted?"_
```
┌──────────┐
│ User │ "Which services unhealthy?"
└────┬─────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) - Multi-step reasoning │
│ │
│ STEP 1: [Thought] Need to check all services │
│ [Decision] Use list_services tool │
│ [Action] list_services() │
└────┬───────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Tool: list_services → Portainer API │
│ │
│ Returns: [ │
│ {"name": "core-api", "status": "running"}, │
│ {"name": "jellyfin", "status": "running"}, │
│ {"name": "uptime-kuma", "status": "running"}, │
│ ... │
│ ] │
└────┬───────────────────────────────────────────────────────────┘
│
│ Result → Agent
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ STEP 2: [Observation] All services show "running" │
│ [Thought] Need health check details from monitoring │
│ [Decision] Use check_service_health for each │
│ [Action] Loop through services │
└────┬───────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Tool: check_service_health (for each service) │
│ │
│ check_service_health("core-api") │
│ → Uptime Kuma API → {"status": "up", "ping": "23ms"} │
│ │
│ check_service_health("jellyfin") │
│ → Uptime Kuma API → {"status": "down", "ping": "timeout"} │
│ │
│ check_service_health("uptime-kuma") │
│ → Uptime Kuma API → {"status": "up", "ping": "5ms"} │
└────┬───────────────────────────────────────────────────────────┘
│
│ Results → Agent
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ STEP 3: [Observation] Jellyfin is down! │
│ [Thought] User asked which need restarting │
│ [Decision] Report findings │
│ [Action] Generate response with recommendation │
└────┬───────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Ollama (mistral:7b) - Final synthesis │
│ │
│ "Based on health checks, Jellyfin (media.schweitz.net) is │
│ currently unhealthy and not responding to health probes. │
│ │
│ Recommendation: Restart the jellyfin service. │
│ │
│ Would you like me to restart it for you?" │
└────┬───────────────────────────────────────────────────────────┘
│
│ SSE stream with multiple status updates:
│ [💭 Analyzing...]
│ → [🔧 Listing services...]
│ → [✓ Found 15 services]
│ → [🔧 Checking health...]
│ → [✓ Checked 15 monitors]
│ → Response
│
▼
┌──────────┐
│ User │
└──────────┘
Multi-Tool Flow:
═══════════════
┌─────────────────┐
│ Agent Reasoning │
│ (mistral:7b) │
└────┬────────────┘
│
┌────▼─────────────────────────────────┐
│ ReAct Loop (LangGraph) │
│ │
│ Thought → Action → Observation │
│ ↓ ↓ ↑ │
│ Analyze Execute Process │
│ Tool Result │
└──────────────────────────────────────┘
│
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│ Tool 1 │ │ Tool 2 │ │ Tool 3 │
│ list_ │ │ check_ │ │ check_ │
│services │ │ health │ │ health │
│ │ │ (x15) │ │ ... │
└─────────┘ └─────────┘ └─────────┘
│ │ │
┌────▼────────────▼────────────▼────┐
│ External Systems │
│ • Portainer API │
│ • Uptime Kuma API │
└───────────────────────────────────┘
```
**Models Used**:
- `mistral:7b` (all reasoning, tool orchestration, synthesis)
**Tool Call Sequence**:
1. `list_services()` → Portainer → 15 services
2. Loop: `check_service_health(service)` × 15 → Uptime Kuma
3. Analyze results → Identify unhealthy
4. Synthesize recommendation
**Why Single Model Works**:
- mistral:7b maintains context across tool calls
- LangGraph manages the ReAct loop state
- Agent "thinks" between each tool call
- No model switching needed for multi-step reasoning
---
## Model Selection Summary
### Current Implementation:
| Scenario | Model Used | Reason |
|----------|-----------|--------|
| **Agent mode** (any query) | `mistral:7b` | Supports tool calling |
| **Direct chat** () | User's choice | gemma:2b, gemma:7b, etc. |
| **Embeddings** | `nomic-embed-text` (via Ollama) | No local PyTorch needed |
### Why mistral:7b for Agent?
✅ **Supports tool calling** - Gemma/Gemma2 do not
✅ **Good reasoning** - Handles multi-step logic
✅ **Fast enough** - 7B parameters, ~2-5s responses
✅ **Available locally** - Already in Ollama
### Future Enhancements:
```
┌────────────────────────────────────────────┐
│ Potential Model Routing │
│ │
│ Task Type → Model │
│ ──────────────────────────────────── │
│ General reasoning → mistral:7b │
│ Code generation → codestral:latest │
│ Fast queries → gemma:2b │
│ Complex analysis → mixtral:8x7b │
│ Embeddings → nomic-embed-text │
└────────────────────────────────────────────┘
```
Could implement model routing in agent:
- Detect task type (code vs general vs analysis)
- Route to specialized model
- Return to mistral:7b for synthesis
---
## Component Communication Matrix
```
Core API Components
═══════════════════
┌─────────────┬──────────┬────────┬────────┬─────────┐
│ Component │ Mistral │ Ollama │ Tools │ External│
│ │ :7b │ API │ │ APIs │
├─────────────┼──────────┼────────┼────────┼─────────┤
│ AI │ │ ✓ │ │ │
│ Controller │ Routes │ Direct │ │ │
│ │ │ call │ │ │
├─────────────┼──────────┼────────┼────────┼─────────┤
│ Unified │ ✓ │ ✓ │ ✓ │ │
│ Agent │ Reasoning│ LLM │ Calls │ │
│ │ │ invoke │ │ │
├─────────────┼──────────┼────────┼────────┼─────────┤
│ Tools │ │ │ │ ✓ │
│ │ │ │ │ Portainer│
│ │ │ │ │ NPM, Kuma│
├─────────────┼──────────┼────────┼────────┼─────────┤
│ SSE │ │ │ ✓ │ │
│ Formatter │ │ │ Status │ │
│ │ │ │ events │ │
└─────────────┴──────────┴────────┴────────┴─────────┘
Legend:
═══════
✓ = Direct communication
Routes = Decision point, passes through
```
---
## Performance Characteristics
### Response Times (Typical):
| Scenario | Time to First Token | Total Time | Model Calls |
|----------|---------------------|------------|-------------|
| **Knowledge query** | ~500ms | 2-3s | 1 (mistral:7b) |
| **Single tool use** | ~500ms | 4-6s | 2 (reasoning + synthesis) |
| **Multi-tool query** | ~500ms | 8-15s | 3+ (reasoning per tool + synthesis) |
| **Code generation** | ~500ms | 5-10s | 2 (read docs + generate) |
### Streaming Benefits:
```
Without Streaming:
User waits → → → [silence] → → → Full response
With Streaming:
User sees → [💭 Thinking] → [🔧 Tool use] → [✓ Done] → Response chunks
↑ 500ms ↑ 2s ↑ 4s
```
User perceives faster response due to immediate feedback!
---
## Key Architectural Decisions
### ✅ Single Agent Model (mistral:7b)
**Pro**: Maintains context across tool calls, simpler architecture
**Con**: Can't leverage specialized models for specific tasks
### ✅ Ollama-Based Embeddings
**Pro**: No local PyTorch (~2GB saved), flexible model switching
**Con**: Network dependency on Ollama service
### ✅ OpenAI-Compatible API
**Pro**: Works with any OpenAI client, easy integration
**Con**: Must convert between formats
### ✅ Tool-Based Architecture
**Pro**: Extensible, clear separation of concerns
**Con**: Each tool call adds latency
### ✅ Streaming with Status Indicators
**Pro**: Transparent reasoning, better UX
**Con**: More complex implementation
---
## Future Optimizations
### 1. Model Routing
Add intelligence to route requests to specialized models:
- Code → `codestral:latest`
- Analysis → `mixtral:8x7b`
- Fast queries → `gemma:2b`
### 2. Tool Result Caching
Cache frequently-accessed infrastructure data:
- Service list (60s TTL)
- Domain list (5min TTL)
- Reduces tool call latency
### 3. Parallel Tool Execution
When independent tools needed:
```python
results = await asyncio.gather(
check_service_health("service1"),
check_service_health("service2"),
check_service_health("service3"),
)
```
Reduces 3×2s = 6s to ~2s
### 4. Smaller Agent Model
Try `gemma2:9b` or `qwen2.5:7b` if they support tools:
- Potentially faster inference
- Lower memory usage
---
## Conclusion
The unified agent architecture successfully:
- ✅ Routes all requests through single intelligent orchestrator
- ✅ Uses `mistral:7b` for tool-calling capability
- ✅ Maintains transparent reasoning via streaming
- ✅ Integrates with existing infrastructure (Portainer, NPM, Kuma)
- ✅ Works with any OpenAI-compatible client
- ✅ Saves ~2GB memory by using Ollama embeddings
Next steps: Test with Open WebUI and document usage for end users.
docs/sessions/2025-11-23-model-level-routing.md
-347
View File
@@ -1,347 +0,0 @@
# Migration to Model-Level Tool Routing
**Date**: 2025-11-23
**Status**: Complete
**Impact**: Simplified architecture, LLM decides tool usage
## Summary
Removed application-level routing (`use_agent` parameter) in favor of model-level routing where mistral:7b autonomously decides whether to use tools or answer directly.
## Architectural Change
### Before (Application-Level Routing):
```python
# AI Controller decides routing
if request.use_agent:
→ Route to agent (mistral:7b with tools)
else:
→ Direct Ollama call (any model)
```
**Problem**: Application layer must decide which queries need tools
### After (Model-Level Routing):
```python
# Always route through agent, LLM decides tool usage
→ Unified Agent (mistral:7b with tools)
→ LLM analyzes query autonomously
→ LLM decides: use tools OR answer directly
```
**Solution**: LLM understands context and decides intelligently
## Why This is Better
### ✅ LLM Already Has This Capability
LangGraph's `create_react_agent` means:
- mistral:7b sees available tools during generation
- mistral:7b outputs tool calls when needed
- mistral:7b answers directly when tools aren't needed
- **No application-level classification required**
### ✅ Simpler Code
**Removed**:
- `use_agent: bool` parameter from request schema
- Conditional routing logic in ai_controller.py
- Need to document when to use `use_agent=true`
**Result**: Single code path for all requests
### ✅ More Intelligent
The LLM understands nuance better than boolean flags:
| Query | LLM Decision | Application Would Have |
|-------|--------------|------------------------|
| "What is Docker?" | Answer directly (no tools) | ❌ Might route wrong |
| "Is core-api running?" | Use tool (needs real data) | ✅ Correct |
| "List services and explain what Docker is" | Use tool + knowledge | ✅ Handles complexity |
### ✅ Consistent UX
- Always get thinking indicators `[💭 Analyzing...]`
- Always see tool usage `[🔧 Checking services...]`
- More transparent reasoning process
### ✅ Perfect for Homelab Context
- **Token usage doesn't matter** - Running locally on Ollama (free)
- **Latency increase minimal** - ~1-2s extra for simple queries
- **Flexibility matters more** - Edge cases handled automatically
## Implementation Changes
### 1. Removed `use_agent` Parameter
**File**: [src/api/v1/schemas.py](../../services/core-api/src/api/v1/schemas.py:44)
```python
# REMOVED
use_agent: bool = Field(
default=True,
description="Use intelligent agent with tool calling and reasoning (recommended)"
)
```
Now all requests go through agent by default.
### 2. Simplified AI Controller
**File**: [src/controllers/ai_controller.py](../../services/core-api/src/controllers/ai_controller.py:307-373)
```python
# Before
if request.use_agent and AGENT_AVAILABLE:
# Route to agent
else:
# Direct Ollama
# After
if AGENT_AVAILABLE:
try:
# Always route through agent
# mistral:7b decides tool usage
except Exception as e:
# Fallback to direct Ollama if agent fails
```
Added try-except for graceful fallback if agent initialization fails.
### 3. Maintained Fallback
If agent is unavailable or fails:
- Falls back to direct Ollama call
- Uses requested model (gemma:2b, gemma:7b, etc.)
- No intelligent tool routing, just basic chat
## How It Works
### LangGraph ReAct Loop
```
User Query
↓
mistral:7b (with bound tools)
↓
[Thought] Analyze query + available tools
↓
[Decision] Does this need a tool?
├─→ NO → Generate answer directly
└─→ YES → Call tool(s) → Get results → Synthesize answer
```
The model sees tool descriptions and autonomously decides:
```python
# Tools are bound to the LLM
llm_with_tools = ChatOllama(model="mistral:7b").bind_tools(tools)
# LLM output contains tool_calls if it wants to use tools
response = llm_with_tools.invoke(messages)
if response.tool_calls:
# Execute tools
else:
# Return answer directly
```
**Key Point**: The application doesn't decide tool usage - it just checks if the LLM outputted tool calls.
## Test Results
All query types work correctly with mistral:7b deciding autonomously:
### Test 1: Simple Math (No Tools)
```json
Query: "What is 2+2?"
Response: "The sum of 2+2 is 4."
Tool Calls: None ✓
Time: ~2s
```
### Test 2: Infrastructure Query (Needs Tools)
```json
Query: "List all running services"
Response: [Detailed service list with ports]
Tool Calls: list_services ✓
Time: ~5s
```
### Test 3: Knowledge Question (No Tools)
```json
Query: "What is Docker?"
Response: [Detailed Docker explanation]
Tool Calls: None ✓
Time: ~2s
```
### Test 4: Streaming with Tools
```
Query: "Check service health for core-api"
Stream: [💭 Analyzing...] → "To check the health status..."
Tool Calls: check_service_health ✓
Time: ~4s
```
## Performance Impact
### Latency Comparison
| Query Type | Before (use_agent=false) | After (always agent) | Delta |
|------------|-------------------------|---------------------|-------|
| Simple math | ~1s (gemma:2b direct) | ~2s (mistral:7b) | +1s |
| Knowledge | ~1-2s (gemma:7b direct) | ~2s (mistral:7b) | ~0s |
| Tool needed | ~5s (mistral:7b agent) | ~5s (mistral:7b) | 0s |
| Multi-tool | ~10s (mistral:7b agent) | ~10s (mistral:7b) | 0s |
**Verdict**: Minimal impact (<2s for simple queries), acceptable for homelab use
### Token Usage
- Agent adds reasoning tokens (~100-200 extra per request)
- **Impact**: Zero (local Ollama, tokens are free)
### Memory Usage
- Consistent: Always uses mistral:7b (~4GB when loaded)
- Before: Mixed (gemma:2b ~1GB, gemma:7b ~3GB, mistral:7b ~4GB)
- **Result**: More predictable resource usage
## Benefits Summary
| Aspect | Benefit |
|--------|---------|
| **Code Complexity** | Reduced - single code path |
| **Maintainability** | Improved - less conditional logic |
| **Flexibility** | Increased - LLM handles edge cases |
| **User Experience** | Consistent - always see reasoning |
| **Performance** | Acceptable - ~1-2s increase for simple queries |
| **Context Awareness** | Better - LLM understands nuance |
## OpenAI Compatibility
Still fully compatible with OpenAI clients:
```bash
# Works with any OpenAI-compatible client
curl -X POST http://api.schweitz.net/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [{"role": "user", "content": "List services"}],
"stream": true
}'
```
**No `use_agent` parameter needed** - agent is transparent to client
## Migration for Clients
### Before
```python
# Client had to know when to use agent
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "List services"}],
extra_body={"use_agent": True} # Had to specify
)
```
### After
```python
# Client doesn't need to know about agent
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "List services"}]
# Agent automatically handles everything
)
```
**Migration**: Remove `use_agent` parameter from client code - it's ignored now
## Fallback Behavior
If agent fails to initialize or encounters an error:
```python
try:
# Route through agent
response = await agent.chat(...)
except Exception as e:
logger.error(f"Agent failed, falling back to direct Ollama: {e}")
# Fall through to direct Ollama call
# Uses requested model without tool capabilities
```
Ensures service remains available even if agent has issues.
## Research Findings
From LangChain/LangGraph best practices:
1. **Tool calling is model-level** - LLMs natively support tool calling, application should just expose tools
2. **ReAct pattern** - LangGraph's `create_react_agent` implements Reason+Act loop where LLM decides actions
3. **Simpler is better** - Industry consensus is to let LLM decide tool usage rather than hardcode routing
4. **`bind_tools()` vs routing** - Use `bind_tools()` for flexibility, use routing only when needed (cost, latency critical)
For homelab context where tokens are free and flexibility matters, model-level routing is the clear winner.
## Future Enhancements
### 1. Model Routing (Optional)
Could add intelligent model selection:
```python
# Agent detects task type
if task_type == "code":
use codestral:latest
elif task_type == "analysis":
use mixtral:8x7b
else:
use mistral:7b (default)
```
### 2. Tool Result Caching
Cache infrastructure queries:
- Service list (60s TTL)
- Domain list (5min TTL)
- Reduces repeated tool calls
### 3. Parallel Tool Execution
When agent needs multiple independent tools:
```python
# Sequential: 3 tools × 2s = 6s
# Parallel: max(tool times) = ~2s
```
## Documentation Updates Needed
- [ ] Update API documentation to remove `use_agent`
- [ ] Update Open WebUI integration guide
- [ ] Add architecture diagrams showing model-level routing
- [x] Document test results and performance characteristics
## Conclusion
**Migration successful!** The system now:
- ✅ Uses model-level routing (LLM decides tool usage)
- ✅ Simpler codebase (removed `use_agent` parameter)
- ✅ More intelligent (LLM understands context)
- ✅ Consistent UX (always see reasoning)
- ✅ Maintains fallback (direct Ollama if agent fails)
- ✅ OpenAI-compatible (clients don't need to change)
The agent is now transparent to users - they just chat naturally and mistral:7b intelligently decides when to use tools.
## Related Files
- [AI Controller](../../services/core-api/src/controllers/ai_controller.py) - Simplified routing
- [Request Schema](../../services/core-api/src/api/v1/schemas.py) - Removed `use_agent`
- [Agent Orchestrator](../../services/core-api/src/agent/orchestrator.py) - Unchanged (already did model-level)
- [Agent Flow Diagrams](../architecture/agent-flow-diagrams.md) - Visual architecture
docs/sessions/2025-11-23-ollama-embeddings-migration.md
-179
View File
@@ -1,179 +0,0 @@
# Migration to Ollama-Based Embeddings
**Date**: 2025-11-23
**Status**: Complete
**Impact**: Removes 2GB+ of dependencies (PyTorch, sentence-transformers)
## Summary
Migrated the Core API embedding system from local `sentence-transformers` models to Ollama's embedding API. This eliminates heavy ML dependencies while providing better performance and flexibility.
## Changes Made
### 1. New Ollama Embedding Client
**File**: [src/models/embeddings_ollama.py](../../services/core-api/src/models/embeddings_ollama.py)
- Created async Ollama-based embedding client
- Uses Ollama's `/api/embeddings` endpoint
- Compatible with existing embedding interface
- No local model loading required
### 2. Updated Qdrant Memory Integration
**File**: [src/memory/qdrant_memory.py](../../services/core-api/src/memory/qdrant_memory.py)
- Changed import from `src.models.embeddings` to `src.models.embeddings_ollama`
- Updated embed calls to use async (`await self.embedding_client.embed_text()`)
- No other changes needed - interface remains the same
### 3. Updated Dependencies
**File**: [services/core-api/requirements.txt](../../services/core-api/requirements.txt)
**Removed**:
```python
sentence-transformers==3.3.1 # ~2GB with PyTorch
```
**Kept**:
```python
qdrant-client==1.11.3 # Still needed for vector storage
```
### 4. Updated Configuration
**File**: [src/config.py](../../services/core-api/src/config.py)
```python
# Old (sentence-transformers):
embedding_model: str = "sentence-transformers/all-MiniLM-L6-v2"
embedding_dimension: int = 384
# New (Ollama):
embedding_model: str = "nomic-embed-text" # Ollama model
embedding_dimension: int = 768 # nomic-embed-text dimension
```
## Benefits
### Memory Savings
- **Before**: ~2-4GB for PyTorch + sentence-transformers
- **After**: ~50MB for qdrant-client only
- **Reduction**: ~95% memory usage reduction
### Deployment Benefits
1. **Faster startup**: No model loading on container start
2. **Smaller image**: Reduced from 8.8GB to ~2GB
3. **Flexibility**: Can switch embedding models in Ollama without code changes
4. **Consistency**: Same embedding model can be used across all services
### Performance
- **Ollama embeddings**: ~10-50ms per text (depending on length)
- **Cached in Ollama**: Faster for repeated texts
- **GPU acceleration**: Ollama uses GPU if available
- **No cold start**: Ollama keeps model loaded
## Ollama Embedding Models
The system now uses `nomic-embed-text` by default (768 dimensions). Other options:
| Model | Dimensions | Use Case |
|-------|-----------|----------|
| `nomic-embed-text` | 768 | General purpose (default) |
| `mxbai-embed-large` | 1024 | High quality embeddings |
| `all-minilm` | 384 | Faster, smaller embeddings |
To change: Update `embedding_model` and `embedding_dimension` in settings or env vars.
## Migration Steps
For clean deployment after this change:
1. **Delete persisted venv** (to reinstall without sentence-transformers):
```bash
rm -rf /home/jpmschweitzer/docker-data/core-api/venv
```
2. **Ensure Ollama has embedding model**:
```bash
docker exec ollama ollama pull nomic-embed-text
```
3. **Restart Core API stack** in Portainer
- Will reinstall dependencies from updated requirements.txt
- First startup may take 2-3 minutes for pip install
4. **Verify embeddings work**:
```bash
curl -X POST http://192.168.86.149:8083/v1/embeddings \
-H "Content-Type: application/json" \
-d '{"input": "test text"}'
```
## Backward Compatibility
### Existing Qdrant Collections
- **No migration needed**: Vector dimensions match
- If using `all-MiniLM-L6-v2` (384d): Change to `all-minilm` in Ollama
- If changing dimensions: Need to recreate Qdrant collections
### Old Embedding Client
- Keep `src/models/embeddings.py` for now (not used)
- Can be removed in future cleanup
- No imports reference it after migration
## Rollback Plan
If issues occur, revert by:
1. Change import back in `qdrant_memory.py`:
```python
from src.models.embeddings import get_embedding_client
```
2. Add back to requirements.txt:
```python
sentence-transformers==3.3.1
```
3. Revert config.py model name
4. Delete venv and restart
## Testing
### Test Embedding Generation
```python
from src.models.embeddings_ollama import get_embedding_client
client = get_embedding_client()
embedding = await client.embed_text("hello world")
print(f"Dimension: {len(embedding)}") # Should be 768
```
### Test Qdrant Integration
```python
from src.memory.qdrant_memory import get_qdrant_memory
from src.memory.schemas import ConversationTurn, MessageRole
from datetime import datetime
memory = get_qdrant_memory()
turn = ConversationTurn(
role=MessageRole.USER,
content="Test message",
timestamp=datetime.now(),
turn_number=1
)
await memory.add_turn("test-conv-123", turn) # Should work
```
## Notes
- Ollama must be running and accessible at `OLLAMA_BASE_URL`
- Embedding model must be pulled in Ollama before first use
- Memory system will be implemented in Phase 2 - this prepares the foundation
- Agent framework (LangChain) still included for unified agent implementation
## Related Changes
- Stack memory limit updated from 2G to 6G (for agent framework burst needs)
- Memory reservation updated from 512M to 1G (baseline usage)
- Agent implementation using LangGraph (separate work)
- Agent now uses `mistral:7b` (tool-calling capable) instead of `gemma:7b`
docs/sessions/2025-11-24-lightweight-model-testing.md
-307
View File
@@ -1,307 +0,0 @@
# Lightweight Model Testing for Tool Calling
**Date**: 2025-11-24
**Tested By**: Claude Code
**Objective**: Evaluate lighter models (gemma3-tools:1b, phi3:mini) as potential replacements for mistral:7b in the agent orchestrator
## Executive Summary
**Recommendation**: **Continue using mistral:7b** for the agent orchestrator.
While `gemma3-tools:1b` demonstrates basic tool calling capability, it has reliability issues with tool selection that make it unsuitable for production use. The `phi3:mini` model does not support tool calling at all.
## Test Setup
### Models Tested
- **gemma3-tools:1b** (999MB) - Tool-capable variant
- **gemma3:4b** (4.3B) - Regular variant (NO tool support)
- **gemma3:12b** (12.2B) - Larger variant (NO tool support)
- **phi3:mini** (3.8B) - General purpose model (NO tool support)
- **mistral:7b** (7.2B) - Current production model (reference)
### Test Framework
Direct Ollama API calls using OpenAI function calling format:
- 3 tools defined: `list_services`, `get_service_details`, `web_search`
- 3 test scenarios: conversation, simple tool use, parameterized tool use
### Test Cases
| Test Case | Description | Expected Behavior |
|-----------|-------------|-------------------|
| **Simple Conversation** | "Hello, how are you?" | No tool use, conversational response |
| **Service Listing** | "Can you list all the running services?" | Call `list_services` tool |
| **Service Details** | "Tell me about the ollama service" | Call `get_service_details` with arg `service_name="ollama"` |
## Test Results
### gemma3-tools:1b Results
| Test Case | Result | Notes |
|-----------|--------|-------|
| **Simple Conversation** | ✅ PASS | Correctly responded without tools |
| **Service Listing** | ❌ FAIL | No tool called; returned raw JSON schema instead |
| **Service Details** | ⚠️ PARTIAL | Called `list_services` instead of `get_service_details` |
**Score**: 1/3 tests passed
**Issues Identified**:
1. **Inconsistent tool calling**: Sometimes calls tools, sometimes doesn't
2. **Wrong tool selection**: Called `list_services` when `get_service_details` was more appropriate
3. **Erratic responses**: Sometimes outputs raw JSON schema instead of calling tools
**Example Problem Response**:
```json
{
"content": "{\"type\": \"function\", \"function\": {\"name\":\"list_services\",..."
}
```
Instead of actually calling the tool, it returned the tool definition as text.
### gemma3:4b Results
| Test Case | Result | Notes |
|-----------|--------|-------|
| **All Tests** | ❌ FAIL | HTTP 400: "does not support tools" |
**Score**: 0/4 tests passed
**Conclusion**: `gemma3:4b` (regular variant) has **NO tool support**. Only the `gemma3-tools:1b` variant includes tool calling capabilities.
### gemma3:12b Results
| Test Case | Result | Notes |
|-----------|--------|-------|
| **All Tests** | ❌ FAIL | HTTP 400: "does not support tools" |
**Score**: 0/4 tests passed
**Conclusion**: `gemma3:12b` (regular variant) has **NO tool support**. Despite being larger than mistral:7b (12.2GB vs 7.2GB), it lacks tool calling architecture.
### phi3:mini Results
| Test Case | Result | Notes |
|-----------|--------|-------|
| **All Tests** | ❌ FAIL | HTTP 400: "does not support tools" |
**Score**: 0/3 tests passed
**Conclusion**: `phi3:mini` has **no tool calling support** in Ollama. The model architecture or quantization does not include tool calling capabilities.
### mistral:7b Results (Reference)
| Test Case | Result | Notes |
|-----------|--------|-------|
| **Simple Conversation** | ✅ PASS | Clean conversational response |
| **Service Listing** | ✅ PASS | Successfully called `list_services` |
| **Service Details** | ✅ PASS | Successfully called appropriate tool |
**Score**: 3/3 tests passed
## Analysis
### Why gemma3-tools:1b Fails
Despite being marketed as a "tools" variant, `gemma3-tools:1b` has fundamental issues:
1. **Training Instability at 1B Scale**: Tool calling requires understanding complex JSON schemas and function signatures. At 1B parameters, the model lacks the capacity for reliable tool orchestration.
2. **Format Confusion**: The model sometimes confuses:
- **Tool definition** (JSON schema of available tools)
- **Tool invocation** (actually calling a tool with arguments)
- **Tool response** (the result returned by a tool)
3. **Insufficient Context Window**: With tools, the context includes:
- System prompt (~200 tokens)
- Tool definitions (~300 tokens per tool)
- Conversation history
- User message
A 1B model struggles to maintain coherent reasoning across this context.
### Why mistral:7b Works Well
1. **7B parameter scale** provides sufficient capacity for:
- Understanding tool schemas
- Reasoning about which tool to use
- Formatting tool calls correctly
- Synthesizing tool results into natural responses
2. **Trained specifically for tool/function calling** with Mistral's instruction-following architecture
3. **Proven in production** - LangChain/LangGraph documentation uses mistral:7b as a reference model for agents
## Performance Comparison
| Metric | gemma3-tools:1b | gemma3:4b | gemma3:12b | mistral:7b |
|--------|-----------------|-----------|------------|------------|
| **Model Size** | 999MB | 4.3GB | 12.2GB | 7.2GB |
| **Tool Support** | ⚠️ Yes (unreliable) | ❌ No | ❌ No | ✅ Yes |
| **Memory Usage** | ~1.5GB | ~5GB | ~13GB | ~8GB |
| **Inference Speed** | ~300ms | ~600ms | ~1200ms | ~800ms |
| **Tool Reliability** | ⚠️ 33% | N/A | N/A | ✅ 100% |
| **Tool Selection** | ⚠️ Low | N/A | N/A | ✅ High |
| **Production Ready** | ❌ No | ❌ No | ❌ No | ✅ Yes |
**Key Finding**: Only the `-tools` variant of gemma3 supports tool calling. Regular gemma3 models (4b, 12b) do NOT have tool support, regardless of size.
## Why Size Doesn't Matter Here
In a cloud/API context, you'd want the smallest model possible to reduce costs. But in our homelab:
### Our Context:
- **Free inference** (running locally on Ollama)
- **GPU available** (RTX 2080 Ti with 11GB VRAM)
- **Single user** (no concurrent load)
- **Quality > Speed** (correctness matters more than 500ms latency)
### Trade-off Analysis:
```
gemma3-tools:1b savings:
- Memory: 6.5GB saved (we have 11GB available, not constrained)
- Speed: 500ms faster (2s → 1.5s, marginal UX improvement)
- Cost: $0 saved (local inference is already free)
mistral:7b benefits:
- Reliability: 100% vs 33% success rate (CRITICAL)
- Tool selection: Correct tool vs wrong tool
- Response quality: Natural synthesis vs confused output
```
**Conclusion**: The savings don't justify the reliability loss.
## Integration Test Results
### Discovered During Testing
Our current implementation already handles the case correctly:
**File**: [services/core-api/src/agent/orchestrator.py](../../services/core-api/src/agent/orchestrator.py:36-40)
```python
# The agent model must support tool calling
self.llm = ChatOllama(
model=self.settings.agent_model, # mistral:7b
base_url=self.settings.ollama_base_url,
temperature=0.7,
)
```
The agent is hardcoded to use `agent_model` from config (currently `mistral:7b`). This is correct because:
1. **Tool calling is a requirement** - The agent uses `create_react_agent` which requires tool support
2. **Not all models support tools** - As demonstrated by phi3:mini
3. **Quality matters** - gemma3-tools:1b technically works but unreliably
## Recommendations
### Short Term (Current Implementation) ✅
**Keep using mistral:7b** for the agent orchestrator:
- Proven reliability
- Excellent tool calling support
- No resource constraints in homelab environment
### Medium Term (Monitoring)
**Watch for**:
- Ollama releases of newer tool-capable models (e.g., `llama3-groq-tool-use`)
- Gemma4 or Phi4 with improved tool calling
- Qwen2.5 variants (some support tools)
**Test criteria for replacement**:
- 100% success rate on tool calling tests
- Correct tool selection (not just "can call tools")
- Consistent response format
- Production-ready error handling
### Long Term (Optimization)
**If memory becomes a constraint**:
1. Test `gemma2:9b` - Larger than 1B, might have better tool support
2. Test `qwen2.5:7b` - Similar size to mistral, different architecture
3. Consider quantization of mistral:7b (Q4 or Q5) to reduce memory footprint
**If latency becomes critical**:
1. Upgrade GPU (RTX 4070+ for faster inference)
2. Implement tool result caching (see [agent-flow-diagrams.md](../architecture/agent-flow-diagrams.md#future-optimizations))
3. Use parallel tool execution for multi-tool queries
## Documentation Updates Needed
Based on testing findings:
### 1. Update Agent Flow Diagrams ✅ (In Progress)
**File**: [docs/architecture/agent-flow-diagrams.md](../architecture/agent-flow-diagrams.md)
Remove references to `use_agent` flag (already deprecated, see [model-level-routing.md](./2025-11-23-model-level-routing.md))
### 2. Update Model Recommendations
**Location**: README.md or AGENTS.md
Add section on model requirements:
```markdown
## Agent Model Requirements
The agent orchestrator requires a model with **tool calling support**. Not all models support this feature.
### Tested Models (2025-11-24):
- ✅ **mistral:7b** - Recommended (current production, 100% reliability)
- ⚠️ **gemma3-tools:1b** - Has tool support but unreliable (33% success rate)
- ❌ **gemma3:4b** - Does not support tools
- ❌ **gemma3:12b** - Does not support tools (even though larger than mistral!)
- ❌ **phi3:mini** - Does not support tools
**Important**: Only the `-tools` suffix variants of gemma3 have tool calling. Regular gemma3 models lack this capability.
### Switching Models:
To change the agent model, edit `services/core-api/.env`:
```bash
AGENT_MODEL=mistral:7b
```
```
## Appendix: Raw Test Output
### Test Run 1: Direct Ollama API
```bash
$ python3 test_tool_models.py
################################################################################
# TESTING MODEL: gemma3-tools:1b
################################################################################
Test Case: Simple Conversation (No Tools)
✅ Correctly responded without tools
Response: Hello there! I'm doing well, thank you for asking. How about you?
Test Case: Service Listing (Should Use Tool)
❌ No tool called when it should have been
Response: {"type": "function", "function": {"name":"list_services",...
Test Case: Service Details (Should Use Tool with Args)
⚠️ Wrong tool: expected get_service_details, got list_services
################################################################################
# TESTING MODEL: phi3:mini
################################################################################
All tests: ❌ HTTP 400: "does not support tools"
################################################################################
# TESTING MODEL: mistral:7b
################################################################################
Test Case: Simple Conversation: ✅ PASS
Test Case: Service Listing: ✅ PASS
Test Case: Service Details: ✅ PASS
```
## Conclusion
**Use mistral:7b** for the agent orchestrator. The benefits of a smaller model don't outweigh the reliability issues in our homelab context. Monitor for future model releases that may offer better tool calling at smaller scales.
---
**Status**: Testing complete, documentation updated
**Next Steps**: Clean up `use_agent` references in flow diagrams, update model documentation