# End-to-End API Tests

These tests make real HTTP requests to the running Tatlock API server to verify the complete stack works correctly.

## Prerequisites

1. **Server must be running** on `http://localhost:8000`
2. **Ollama must be running** with `mistral-nemo:latest` model
3. **Redis must be running** (for benchmarking)

## Running the Tests

### Start the server first:

```bash
# Terminal 1: Start the server
uvicorn src.main:app --reload
```

### Run the E2E tests:

```bash
# Terminal 2: Run E2E tests
PYTHONPATH=/mnt/media/Projects/tatlock pytest tests/e2e/ -v
```

### Run specific test categories:

```bash
# Test chat completions only
pytest tests/e2e/test_api_endpoints.py::TestChatCompletionsE2E -v

# Test responses API only
pytest tests/e2e/test_api_endpoints.py::TestResponsesAPIE2E -v

# Test streaming only
pytest tests/e2e/test_api_endpoints.py::TestStreamingE2E -v

# Test Steward integration specifically
pytest tests/e2e/test_api_endpoints.py::TestStewardIntegration -v
```

## What These Tests Verify

### 1. Chat Completions Endpoint (`/v1/chat/completions`)

- ✅ Simple calculations trigger calculator tool
- ✅ Search queries trigger web search
- ✅ Multi-turn conversations maintain context
- ✅ Complex requests use multiple tools
- ✅ Simple greetings don't trigger unnecessary tools
- ✅ Date/time queries trigger datetime tools

### 2. Responses API Endpoint (`/v1/responses`)

- ✅ Reasoning output includes Steward's analysis
- ✅ Multi-turn conversations show in Steward reasoning
- ✅ Response structure follows OpenAI Responses format

### 3. Streaming

- ✅ Chat completions streaming works
- ✅ Steward reasoning appears in stream
- ✅ Proper SSE format with chunks

### 4. Error Handling

- ✅ Invalid model returns 404
- ✅ Missing required fields return 422
- ✅ Invalid parameters return 422

### 5. Steward Integration

- ✅ Steward recommends correct capabilities
- ✅ Steward detects conversation context
- ✅ Steward analysis appears in all responses

## Expected Behavior

When tests run, you should see in the server logs:

```
INFO     creating_response_with_steward
INFO     preprocessing_request
INFO     operation_started             operation=steward_analysis
INFO     steward_analysis_complete     recommended=[...] complexity=simple
INFO     tatlock_run_with_scoped_tools
INFO     tatlock_response_generated
INFO     tool_tracking_finalized
```

## Test Scenarios

### Simple Calculation
```
User: "What is 144 divided by 12?"
Expected: Calculator tool used, answer is "12"
```

### Web Search
```
User: "What is the capital of France?"
Expected: Search may be used, answer mentions "Paris"
```

### Multi-Turn
```
User: "What is 15 times 4?"
Assistant: "60"
User: "Now add 20 to that result."
Expected: Context recognized, answer is "80"
```

### Combined Tools
```
User: "Calculate the square root of 256, then search for what number squared equals that result."
Expected: Both calculator and search recommended
```

### Date/Time
```
User: "What is today's date?"
Expected: Datetime tool used, current date returned
```

## Troubleshooting

### Tests fail with connection error

Make sure the server is running:
```bash
uvicorn src.main:app --reload
```

### Tests timeout

- Check that Ollama is running and responsive
- Increase timeout in test file if needed (default: 60s)

### Tool usage not detected

- Check server logs to see if tools are actually being called
- Verify Steward preprocessing is happening (look for `steward_analysis` logs)

### Inconsistent results

- LLM responses can vary - tests check for key indicators rather than exact text
- If a test occasionally fails, it might be due to LLM variance
- Check the actual response content in the test output

## Coverage

These tests complement the unit and integration tests by:

1. **Testing the full HTTP stack** - Request parsing, routing, middleware
2. **Testing real LLM behavior** - Not mocked, actual Ollama responses
3. **Testing real tool execution** - Calculator, datetime, search actually run
4. **Testing Steward preprocessing** - Real analysis and tool scoping
5. **Testing error handling** - HTTP error codes and error responses

Together with unit/integration tests, this provides comprehensive coverage of the entire system.