portainer-core/services/core-ai/tests

Core-AI Test Suite

Layered testing approach to diagnose and validate the core-ai service.

Quick Start

# Run all tests in sequence
bash tests/run_all_tests.sh

# Or run individual layers
pytest tests/test_01_environment.py -v -s
pytest tests/test_02_litellm_raw.py -v -s
pytest tests/test_03_message_format.py -v -s
pytest tests/test_04_agent.py -v -s
pytest tests/test_05_api.py -v -s  # Requires service running

Test Layers

Layer 1: Environment & Configuration

File: test_01_environment.py

Tests basic configuration and environment setup:

• ✓ Settings load correctly
• ✓ Required environment variables are set
• ✓ Ollama is reachable
• ✓ Target model is available in Ollama
• ✓ System prompt variant exists

When this fails: Check environment variables, Ollama connectivity, model availability

Layer 2: Raw LiteLLM Connection

File: test_02_litellm_raw.py

Tests direct LiteLLM → Ollama communication without any wrappers:

• ✓ Simple completion works
• ✓ System prompt is respected
• ✓ Streaming mode works
• ✓ Can answer "What is the capital of France?"

When this fails: Issue is in LiteLLM/Ollama integration, not the agent wrapper

Layer 3: Message Formatting & Prompts

File: test_03_message_format.py

Tests prompt management and message structure:

• ✓ Prompts are defined correctly
• ✓ System prompt injection works
• ✓ Messages are formatted properly
• ✓ No duplicate system prompts

When this fails: Check prompts.py and message formatting logic

Layer 4: Agent Logic

File: test_04_agent.py

Tests the SimpleLiteLLMAgent class:

• ✓ Agent initializes correctly
• ✓ Streaming chat works
• ✓ Non-streaming completion works
• ✓ System prompt is injected
• ✓ Can answer "What is the capital of France?"

When this fails: Issue is in the agent wrapper (src/agent.py)

Layer 5: API Integration

File: test_05_api.py

Tests the HTTP API endpoints (requires service running):

• ✓ Health check works
• ✓ Non-streaming API works
• ✓ Streaming API works
• ✓ OpenAI-compatible format
• ✓ Error handling

When this fails: Issue is in the API layer (main.py)

Diagnostic Tools

Check Ollama

python diagnostics/check_ollama.py

Quick script to verify:

• Ollama connectivity
• Available models
• Basic text generation

Test LiteLLM Direct

python diagnostics/test_litellm_direct.py

Standalone test that bypasses all abstractions and tests raw LiteLLM → Ollama.

Running Tests

All tests in sequence (recommended)

bash tests/run_all_tests.sh

This runs all layers and stops at the first failure, helping you identify exactly where the issue is.

Individual test layers

# Install dependencies first
pip install -r requirements.txt

# Run specific layer
pytest tests/test_01_environment.py -v -s

With Docker

If running in Docker, exec into the container:

docker exec -it core-ai bash
cd /app
bash tests/run_all_tests.sh

Understanding Test Results

✓ All tests pass

The foundation is solid. If the service still doesn't work, check:

• Application logs
• Request/response formatting
• Client integration

✗ Layer 1 fails

Problem: Environment or configuration issue Fix:

• Check environment variables
• Verify Ollama is running: docker ps | grep ollama
• Check model is available: docker exec ollama ollama list

✗ Layer 2 fails

Problem: LiteLLM/Ollama integration issue Fix:

• Check Ollama logs: docker logs ollama
• Verify model works directly: docker exec ollama ollama run gemma2:9b-instruct-q5_K_M "test"
• Check LiteLLM version compatibility

✗ Layer 3 fails

Problem: Prompt configuration issue Fix:

• Check src/prompts.py has required variants
• Verify SYSTEM_PROMPT_VARIANT env var matches a defined prompt

✗ Layer 4 fails

Problem: Agent wrapper issue Fix:

• Check src/agent.py for bugs
• Review message formatting logic
• Check system prompt injection

✗ Layer 5 fails

Problem: API layer issue Fix:

• Ensure service is running: python main.py
• Check logs for errors
• Verify request/response format

Adding New Tests

Follow the layered approach:

1. Add test to appropriate layer file
2. Use descriptive test names: test_<what_it_tests>
3. Add clear assertions with messages
4. Print useful debug info for when tests pass

Example:

@pytest.mark.asyncio
async def test_new_feature():
    """Test that new feature works"""
    # Setup
    agent = get_simple_litellm_agent()

    # Execute
    result = await agent.some_method()

    # Assert
    assert result is not None, "Result should not be None"
    print(f"✓ Feature works: {result}")

Troubleshooting

Tests hang or timeout

• Increase timeout in test
• Check Ollama is responding: curl http://ollama:11434/api/tags
• Model may be loading on first run (can take 30-60s)

Import errors

pip install -r requirements.txt

Pytest not found

pip install pytest pytest-asyncio

Can't connect to Ollama

• Check docker network: docker network ls
• Verify services are on same network
• Try using IP instead of hostname

Next Steps After Tests Pass

1. Start the service:

   python main.py
   

2. Test manually:

   curl -X POST http://localhost:8086/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"messages": [{"role": "user", "content": "What is the capital of France?"}]}'
   

3. Deploy in Docker:

   docker-compose up core-ai
   

4. Integrate with other services