portainer-core/docs/ADK_Ollama_Research.md

Research on ADK, LiteLLM, and Ollama Integration in core-api

1. Introduction

This document provides a detailed analysis of the AI chat implementation within the core-api service, focusing on the integration of Google's Agent Development Kit (ADK), LiteLLM, and Ollama. The primary goal is to understand the current architecture, identify probable causes for production failures, and propose actionable improvements to enhance stability, maintainability, and performance.

2. Current Implementation Analysis

The core-api service employs a sophisticated but complex dual-path architecture for handling chat completions.

2.1. Dual-Path Architecture

Two distinct endpoints process chat requests:

1. Agent-Based Path (/api/v1/ai/chat/completions): Managed by src/controllers/ai_controller.py, this is the primary, advanced endpoint. It leverages an agent built with the Google ADK for complex logic, including tool usage. If the agent is unavailable or fails, this endpoint critically falls back to the direct path.

2. Direct Ollama Path (/api/v1/chat/completions): Defined in src/api/v1/chat.py, this endpoint provides a simpler, OpenAI-compatible interface that interacts directly with Ollama, bypassing the agent.

This dual-path system, especially the silent fallback in the main controller, creates ambiguity and can mask critical failures in the agent stack.

2.2. ADK and LiteLLM Integration

The core of the agent is in src/agent/orchestrator.py.

• It uses google-adk to define the agent's structure and logic (UnifiedAgent).
• It uses litellm as a compatibility layer to connect the ADK to the Ollama backend. The agent is instantiated on a per-request basis, making it stateless from the ADK's perspective.
• Crucially, the connection to Ollama is configured via the OLLAMA_API_BASE environment variable.

2.3. Configuration Management

Application settings are centralized in src/config.py and loaded from .env files. However, a critical inconsistency exists:

• The direct Ollama client (src/models/ollama_client.py) correctly uses the ollama_base_url setting from the Settings object.
• The ADK/LiteLLM agent (src/agent/orchestrator.py) ignores this and relies exclusively on the OLLAMA_API_BASE environment variable.

This discrepancy is a primary source of configuration fragility.

2.4. Production Environment

The Dockerfile defines the production container. It installs dependencies from requirements.txt (including google-adk and litellm) but does not set the OLLAMA_API_BASE environment variable. This means the agent defaults to LiteLLM's hardcoded http://localhost:11434, which may not be correct in all deployment scenarios.

The Docker HEALTHCHECK only tests the direct Ollama client via /health, meaning the service can report as healthy even if the entire agent stack is non-functional.

3. Potential Causes of Production Errors

The investigation points to several likely causes for the reported failures.

1. Configuration Mismatch (Most Likely Cause): The agent is likely failing because the OLLAMA_API_BASE environment variable is not set or is set incorrectly in the production environment. Because the direct client uses a different configuration variable (ollama_base_url), the fallback mechanism works, and the API returns a successful response, completely hiding the agent's failure. Developers may be unaware that the agent is not being used.

2. Silent Agent Failure: The fallback logic in ai_controller.py prevents any errors from the agent from propagating. While this ensures availability, it makes debugging impossible and hides the fact that advanced features (tool use, complex reasoning) are not executing.

3. Incomplete Health Check: The current health check provides a false sense of security. The service can be "healthy" while the core agent functionality is broken.

4. Suggested Improvements and Optimizations

To address these issues, the following improvements are recommended:

1. Unify Configuration:

   • Action: Refactor src/agent/orchestrator.py to source the Ollama URL from the central Settings object in src/config.py. Remove the dependency on the OLLAMA_API_BASE environment variable.
   • Benefit: Creates a single, unambiguous source of truth for the Ollama URL, simplifying configuration and reducing errors.

2. Eliminate Redundant Endpoint:

   • Action: Deprecate and remove the /api/v1/chat/completions endpoint in src/api/v1/chat.py. The ai_controller should be the sole entry point for all chat-related requests.
   • Benefit: Simplifies the architecture, removes code duplication, and eliminates confusion about which endpoint to use.

3. Improve Health Checks:

   • Action: Implement a dedicated agent health check endpoint (e.g., /health/agent) that specifically invokes the agent and verifies its connection to Ollama via LiteLLM.
   • Benefit: Provides a true signal of the agent's status, enabling reliable automated monitoring and faster failure detection.

4. Introduce an Explicit Failure Mode:

   • Action: Add a configuration flag (e.g., AGENT_FALLBACK_ENABLED) that, when disabled in development/testing environments, causes agent failures to return a 500 error instead of silently falling back.
   • Benefit: Makes debugging the agent significantly easier.

5. Explore Stateful ADK Sessions:

   • Action: Investigate using the ADK's built-in session management (session_service). This would involve creating sessions that persist across multiple requests.
   • Benefit: Could improve performance by reducing agent initialization overhead and would enable more sophisticated, multi-turn conversational memory within the agent's context.

5. Architectural Review and Validity

The current architecture is powerful and ambitious. The use of the Google ADK provides a solid foundation for building advanced, tool-using agents, and the Qdrant-based memory system is robust.

However, its validity is severely undermined by its fragility and opacity. The configuration mismatch and silent fallback mechanism make the system difficult to debug and unreliable in a production setting. The dual-path entry points add unnecessary complexity.

The architecture is fundamentally sound but requires the recommended refactoring to become robust, maintainable, and production-ready. By unifying configuration, improving observability, and simplifying the request flow, the core-api service can reliably deliver on the promise of its advanced agent capabilities.