tatlock/tests/e2e/README.md

End-to-End API Tests

These tests make real HTTP requests to the running Tatlock API server to verify the complete stack works correctly.

Prerequisites

1. Server must be running on http://localhost:8000
2. Ollama must be running with mistral-nemo:latest model
3. Redis must be running (for benchmarking)

Running the Tests

Start the server first:

# Terminal 1: Start the server
uvicorn src.main:app --reload

Run the E2E tests:

# Terminal 2: Run E2E tests
PYTHONPATH=/mnt/media/Projects/tatlock pytest tests/e2e/ -v

Run specific test categories:

# Test chat completions only
pytest tests/e2e/test_api_endpoints.py::TestChatCompletionsE2E -v

# Test responses API only
pytest tests/e2e/test_api_endpoints.py::TestResponsesAPIE2E -v

# Test streaming only
pytest tests/e2e/test_api_endpoints.py::TestStreamingE2E -v

# Test Steward integration specifically
pytest tests/e2e/test_api_endpoints.py::TestStewardIntegration -v

What These Tests Verify

1. Chat Completions Endpoint (/v1/chat/completions)

• ✅ Simple calculations trigger calculator tool
• ✅ Search queries trigger web search
• ✅ Multi-turn conversations maintain context
• ✅ Complex requests use multiple tools
• ✅ Simple greetings don't trigger unnecessary tools
• ✅ Date/time queries trigger datetime tools

2. Responses API Endpoint (/v1/responses)

• ✅ Reasoning output includes Steward's analysis
• ✅ Multi-turn conversations show in Steward reasoning
• ✅ Response structure follows OpenAI Responses format

3. Streaming

• ✅ Chat completions streaming works
• ✅ Steward reasoning appears in stream
• ✅ Proper SSE format with chunks

4. Error Handling

• ✅ Invalid model returns 404
• ✅ Missing required fields return 422
• ✅ Invalid parameters return 422

5. Steward Integration

• ✅ Steward recommends correct capabilities
• ✅ Steward detects conversation context
• ✅ Steward analysis appears in all responses

Expected Behavior

When tests run, you should see in the server logs:

INFO     creating_response_with_steward
INFO     preprocessing_request
INFO     operation_started             operation=steward_analysis
INFO     steward_analysis_complete     recommended=[...] complexity=simple
INFO     tatlock_run_with_scoped_tools
INFO     tatlock_response_generated
INFO     tool_tracking_finalized

Test Scenarios

Simple Calculation

User: "What is 144 divided by 12?"
Expected: Calculator tool used, answer is "12"

Web Search

User: "What is the capital of France?"
Expected: Search may be used, answer mentions "Paris"

Multi-Turn

User: "What is 15 times 4?"
Assistant: "60"
User: "Now add 20 to that result."
Expected: Context recognized, answer is "80"

Combined Tools

User: "Calculate the square root of 256, then search for what number squared equals that result."
Expected: Both calculator and search recommended

Date/Time

User: "What is today's date?"
Expected: Datetime tool used, current date returned

Troubleshooting

Tests fail with connection error

Make sure the server is running:

uvicorn src.main:app --reload

Tests timeout

• Check that Ollama is running and responsive
• Increase timeout in test file if needed (default: 60s)

Tool usage not detected

• Check server logs to see if tools are actually being called
• Verify Steward preprocessing is happening (look for steward_analysis logs)

Inconsistent results

• LLM responses can vary - tests check for key indicators rather than exact text
• If a test occasionally fails, it might be due to LLM variance
• Check the actual response content in the test output

Coverage

These tests complement the unit and integration tests by:

1. Testing the full HTTP stack - Request parsing, routing, middleware
2. Testing real LLM behavior - Not mocked, actual Ollama responses
3. Testing real tool execution - Calculator, datetime, search actually run
4. Testing Steward preprocessing - Real analysis and tool scoping
5. Testing error handling - HTTP error codes and error responses

Together with unit/integration tests, this provides comprehensive coverage of the entire system.