# Core-AI Test Suite

Layered testing approach to diagnose and validate the core-ai service.

## Quick Start

```bash
# Run all tests in sequence
bash tests/run_all_tests.sh

# Or run individual layers
pytest tests/test_01_environment.py -v -s
pytest tests/test_02_litellm_raw.py -v -s
pytest tests/test_03_message_format.py -v -s
pytest tests/test_04_agent.py -v -s
pytest tests/test_05_api.py -v -s  # Requires service running
```

## Test Layers

### Layer 1: Environment & Configuration
**File:** `test_01_environment.py`

Tests basic configuration and environment setup:
- ✓ Settings load correctly
- ✓ Required environment variables are set
- ✓ Ollama is reachable
- ✓ Target model is available in Ollama
- ✓ System prompt variant exists

**When this fails:** Check environment variables, Ollama connectivity, model availability

### Layer 2: Raw LiteLLM Connection
**File:** `test_02_litellm_raw.py`

Tests direct LiteLLM → Ollama communication without any wrappers:
- ✓ Simple completion works
- ✓ System prompt is respected
- ✓ Streaming mode works
- ✓ Can answer "What is the capital of France?"

**When this fails:** Issue is in LiteLLM/Ollama integration, not the agent wrapper

### Layer 3: Message Formatting & Prompts
**File:** `test_03_message_format.py`

Tests prompt management and message structure:
- ✓ Prompts are defined correctly
- ✓ System prompt injection works
- ✓ Messages are formatted properly
- ✓ No duplicate system prompts

**When this fails:** Check prompts.py and message formatting logic

### Layer 4: Agent Logic
**File:** `test_04_agent.py`

Tests the SimpleLiteLLMAgent class:
- ✓ Agent initializes correctly
- ✓ Streaming chat works
- ✓ Non-streaming completion works
- ✓ System prompt is injected
- ✓ Can answer "What is the capital of France?"

**When this fails:** Issue is in the agent wrapper (src/agent.py)

### Layer 5: API Integration
**File:** `test_05_api.py`

Tests the HTTP API endpoints (requires service running):
- ✓ Health check works
- ✓ Non-streaming API works
- ✓ Streaming API works
- ✓ OpenAI-compatible format
- ✓ Error handling

**When this fails:** Issue is in the API layer (main.py)

## Diagnostic Tools

### Check Ollama
```bash
python diagnostics/check_ollama.py
```

Quick script to verify:
- Ollama connectivity
- Available models
- Basic text generation

### Test LiteLLM Direct
```bash
python diagnostics/test_litellm_direct.py
```

Standalone test that bypasses all abstractions and tests raw LiteLLM → Ollama.

## Running Tests

### All tests in sequence (recommended)
```bash
bash tests/run_all_tests.sh
```

This runs all layers and stops at the first failure, helping you identify exactly where the issue is.

### Individual test layers
```bash
# Install dependencies first
pip install -r requirements.txt

# Run specific layer
pytest tests/test_01_environment.py -v -s
```

### With Docker

If running in Docker, exec into the container:
```bash
docker exec -it core-ai bash
cd /app
bash tests/run_all_tests.sh
```

## Understanding Test Results

### ✓ All tests pass
The foundation is solid. If the service still doesn't work, check:
- Application logs
- Request/response formatting
- Client integration

### ✗ Layer 1 fails
**Problem:** Environment or configuration issue
**Fix:**
- Check environment variables
- Verify Ollama is running: `docker ps | grep ollama`
- Check model is available: `docker exec ollama ollama list`

### ✗ Layer 2 fails
**Problem:** LiteLLM/Ollama integration issue
**Fix:**
- Check Ollama logs: `docker logs ollama`
- Verify model works directly: `docker exec ollama ollama run gemma2:9b-instruct-q5_K_M "test"`
- Check LiteLLM version compatibility

### ✗ Layer 3 fails
**Problem:** Prompt configuration issue
**Fix:**
- Check `src/prompts.py` has required variants
- Verify `SYSTEM_PROMPT_VARIANT` env var matches a defined prompt

### ✗ Layer 4 fails
**Problem:** Agent wrapper issue
**Fix:**
- Check `src/agent.py` for bugs
- Review message formatting logic
- Check system prompt injection

### ✗ Layer 5 fails
**Problem:** API layer issue
**Fix:**
- Ensure service is running: `python main.py`
- Check logs for errors
- Verify request/response format

## Adding New Tests

Follow the layered approach:
1. Add test to appropriate layer file
2. Use descriptive test names: `test_<what_it_tests>`
3. Add clear assertions with messages
4. Print useful debug info for when tests pass

Example:
```python
@pytest.mark.asyncio
async def test_new_feature():
    """Test that new feature works"""
    # Setup
    agent = get_simple_litellm_agent()

    # Execute
    result = await agent.some_method()

    # Assert
    assert result is not None, "Result should not be None"
    print(f"✓ Feature works: {result}")
```

## Troubleshooting

### Tests hang or timeout
- Increase timeout in test
- Check Ollama is responding: `curl http://ollama:11434/api/tags`
- Model may be loading on first run (can take 30-60s)

### Import errors
```bash
pip install -r requirements.txt
```

### Pytest not found
```bash
pip install pytest pytest-asyncio
```

### Can't connect to Ollama
- Check docker network: `docker network ls`
- Verify services are on same network
- Try using IP instead of hostname

## Next Steps After Tests Pass

1. **Start the service:**
   ```bash
   python main.py
   ```

2. **Test manually:**
   ```bash
   curl -X POST http://localhost:8086/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"messages": [{"role": "user", "content": "What is the capital of France?"}]}'
   ```

3. **Deploy in Docker:**
   ```bash
   docker-compose up core-ai
   ```

4. **Integrate with other services**