portainer-core/services/core-ai/README.md

Core-AI Service

Simplified AI service for testing LiteLLM → Ollama → Model integration without ADK complexity.

Purpose

This service strips away the ADK layer to isolate and debug the fundamental LiteLLM/Ollama integration. It provides:

• Direct LiteLLM integration - No ADK overhead
• OpenAI-compatible API - Drop-in replacement for testing
• Comprehensive diagnostics - Layered testing to identify issues
• Minimal complexity - Easy to understand and debug

Architecture

HTTP Request → SimpleLiteLLMAgent → LiteLLM → Ollama → Model → Response

Bypassed: Google ADK, tool calling, complex orchestration

Quick Start

1. Install Dependencies

pip install -r requirements.txt

2. Configure Environment

Create .env file or set environment variables:

OLLAMA_BASE_URL=http://ollama:11434
AGENT_MODEL=gemma2:9b-instruct-q5_K_M
SYSTEM_PROMPT_VARIANT=minimal_agent
HOST=0.0.0.0
PORT=8086

3. Run Diagnostics

# Check Ollama connectivity
python diagnostics/check_ollama.py

# Test direct LiteLLM
python diagnostics/test_litellm_direct.py

# Run full test suite
bash tests/run_all_tests.sh

4. Start Service

python main.py

Service will be available at http://localhost:8086

5. Test It

curl -X POST http://localhost:8086/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }'

API Endpoints

GET /health

Health check endpoint

Response:

{
  "status": "ok",
  "service": "core-ai"
}

POST /v1/chat/completions

OpenAI-compatible chat completions endpoint

Request:

{
  "model": "test",
  "messages": [
    {"role": "user", "content": "Your question here"}
  ],
  "stream": false
}

Response (non-streaming):

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "test",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Response here"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}

Streaming: Set "stream": true for Server-Sent Events response

Project Structure

services/core-ai/
├── main.py                 # HTTP server (aiohttp)
├── src/
│   ├── agent.py           # SimpleLiteLLMAgent
│   ├── config.py          # Configuration (Pydantic)
│   ├── prompts.py         # System prompts
│   └── tools.py           # (Unused in this version)
├── diagnostics/
│   ├── check_ollama.py    # Ollama connectivity check
│   └── test_litellm_direct.py  # Direct LiteLLM test
├── tests/
│   ├── test_01_environment.py   # Config tests
│   ├── test_02_litellm_raw.py   # Raw LiteLLM tests
│   ├── test_03_message_format.py # Message formatting
│   ├── test_04_agent.py         # Agent logic tests
│   ├── test_05_api.py           # API endpoint tests
│   ├── run_all_tests.sh         # Run all tests
│   └── README.md                # Test documentation
├── requirements.txt
├── Dockerfile
└── README.md (this file)

Configuration

Configuration is managed via src/config.py using Pydantic Settings.

Environment Variables

Variable	Default	Description
HOST	0.0.0.0	Server host
PORT	8086	Server port
OLLAMA_BASE_URL	http://ollama:11434	Ollama API URL
AGENT_MODEL	gemma2:9b-instruct-q5_K_M	Model name
SYSTEM_PROMPT_VARIANT	minimal_agent	Prompt variant to use
DEBUG	false	Enable debug mode
LOG_LEVEL	INFO	Logging level

Testing

See tests/README.md for comprehensive testing documentation.

Quick test:

bash tests/run_all_tests.sh

This runs 5 layers of tests to isolate issues:

1. Environment & Configuration
2. Raw LiteLLM Connection
3. Message Formatting
4. Agent Logic
5. API Integration

Docker Deployment

Build

docker build -t core-ai:latest .

Run

docker run -d \
  --name core-ai \
  -p 8086:8086 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -e AGENT_MODEL=gemma2:9b-instruct-q5_K_M \
  --network docker-dataplane \
  core-ai:latest

Using Docker Compose

docker-compose -f ../../stacks/core-ai.yml up

Troubleshooting

Service won't start

1. Check logs: docker logs core-ai
2. Verify Ollama is running: docker ps | grep ollama
3. Run diagnostics: python diagnostics/check_ollama.py

No response or timeout

1. Check Ollama logs: docker logs ollama
2. Model may be loading (first run takes 30-60s)
3. Verify model exists: docker exec ollama ollama list
4. Test directly: docker exec ollama ollama run gemma2:9b-instruct-q5_K_M "test"

Wrong or empty responses

1. Check system prompt is loaded (see agent logs)
2. Verify prompt variant exists in src/prompts.py
3. Run Layer 3 tests: pytest tests/test_03_message_format.py -v

Connection refused

1. Check network: docker network inspect docker-dataplane
2. Verify both services are on the same network
3. Try using container IP instead of hostname

Development

Adding New Prompts

Edit src/prompts.py:

PROMPTS = {
    "minimal_agent": "You are a helpful assistant.",
    "my_new_prompt": "Your custom system prompt here."
}

Update environment variable:

SYSTEM_PROMPT_VARIANT=my_new_prompt

Modifying Agent Behavior

Edit src/agent.py - specifically the SimpleLiteLLMAgent class.

Key methods:

• __init__() - Initialization and configuration
• chat() - Streaming chat handler
• chat_completion() - Non-streaming completion handler

Adding Tests

Add to appropriate test layer in tests/:

• Configuration changes → test_01_environment.py
• LiteLLM behavior → test_02_litellm_raw.py
• Message formatting → test_03_message_format.py
• Agent logic → test_04_agent.py
• API changes → test_05_api.py

Comparison with Core-API

Feature	Core-AI	Core-API
ADK Integration	❌ No	✅ Yes
Tool Calling	❌ No	✅ Yes
System Orchestration	❌ No	✅ Yes
Complexity	Low	High
Purpose	Debugging	Production
Direct LiteLLM	✅ Yes	❌ No
Diagnostics	✅ Comprehensive	Limited

Next Steps

If Tests Pass

1. ✅ Foundation is solid
2. Consider migrating fixes to core-api
3. Add ADK layer back in phases
4. Test tool calling integration

If Tests Fail

1. Run diagnostics to identify layer
2. Fix that specific layer
3. Re-run tests
4. Proceed once all pass

Contributing

When making changes:

1. Run diagnostics first
2. Make changes
3. Run full test suite
4. Update relevant documentation
5. Test in Docker environment

License

Part of the tower-of-joy project.