Tatlock - OpenAI-Compatible API with Responses API

A FastAPI-based service providing OpenAI-compatible API endpoints with full Responses API support, reasoning display, and streaming. Features a hybrid architecture with chat completions as a compatibility wrapper around the Responses API.

Current Status

✅ Production-ready testing API with OpenAI Responses API format ✅ Open WebUI integration with reasoning bubbles (<think> tags) ✅ Conversation history with hybrid client/server approach 🚧 PydanticAI integration prepared for future real LLM connection

Architecture Overview

Hybrid API Design

┌─────────────────────────────────────────┐
│  Client (Open WebUI, etc.)              │
└────────┬────────────────────────────────┘
         │
         ├──────────────────────────────────┐
         │                                   │
         v                                   v
┌────────────────────┐          ┌──────────────────────┐
│ /v1/chat/          │ wrapper  │ /v1/responses        │
│ completions        ├─────────>│ (Primary API)        │
│                    │          │                      │
│ • OpenAI compat    │          │ • Reasoning items    │
│ • <think> tags     │          │ • Function calls     │
│ • Legacy support   │          │ • Message items      │
└────────────────────┘          └──────────┬───────────┘
                                           │
                                           v
                                ┌──────────────────────┐
                                │  Agent Interface     │
                                │                      │
                                │ • lorem-tester       │
                                │ • tatlock (future)   │
                                └──────────────────────┘

Key Architectural Decisions:

• Single Source of Truth: Responses API handles all generation logic
• Chat Completions Wrapper: Converts Responses output to Chat format with <think> tags
• Agent Interface: Clean abstraction for multiple models (mock and real)
• Hybrid History: Client sends full context, server optionally tracks conversations

Features

Core API

• ✅ Responses API (/v1/responses) - Primary endpoint with structured output
  • Reasoning items (thinking/extended thinking)
  • Function call items (tool execution)
  • Message items (assistant responses)
  • Streaming and non-streaming modes
• ✅ Chat Completions API (/v1/chat/completions) - Compatibility wrapper
  • Converts reasoning to <think> tags for Open WebUI
  • Maintains OpenAI-compatible format
  • Wraps Responses API (single source of truth)
• ✅ Models API (/v1/models) - Lists available models

Advanced Features

• ✅ Conversation History Management
  • Hybrid approach: client maintains state, server tracks optionally
  • Auto-generated conversation IDs from first message hash
  • Configurable max turns (default: 20)
  • Placeholder for future vector memory (Qdrant)
• ✅ Context Window Management
  • Approximate token counting (~4 chars/token)
  • Context trimming to fit model limits
  • Token usage statistics
• ✅ Parameter Validation
  • Temperature: 0.0-2.0
  • Reasoning effort: none, minimal, low, medium, high, xhigh
  • Max output tokens enforcement
  • Stop sequences (up to 4)
• ✅ Stop Sequence Detection
  • Real-time detection during streaming
  • Stops generation immediately when encountered
• ✅ Max Tokens Enforcement
  • Real-time token counting during streaming
  • Stops when limit reached

Testing Models

• ✅ lorem-tester - Full-featured mock agent
  • Realistic reasoning summaries
  • Random tool/function call generation
  • Error triggers for testing (rate_limit, context_overflow)
  • Temperature variation
• ✅ tatlock - Placeholder for real PydanticAI agent

Open WebUI Integration

• ✅ Reasoning Display - Thinking bubbles shown separately from responses
• ✅ Streaming Support - Smooth word-by-word streaming
• ✅ Error Handling - Graceful error display
• ✅ Model Selection - Both models available in dropdown

Components

• FastAPI: High-performance web framework
• SSE-Starlette: Server-Sent Events for streaming
• Pydantic: Type-safe request/response validation
• Agent Interface: Abstraction for multiple model backends
• Conversation History: Server-side tracking with hybrid approach
• Context Window: Token management and trimming

Requirements

• Python 3.12+ (Python 3.12.11 recommended)
• No external dependencies for mock API
• (Future: Network access for PydanticAI integration)

Installation

1. Clone the repository

git clone <repository-url>
cd tatlock

2. Create a virtual environment

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

4. Configure environment (Optional)

Create a .env file for custom configuration:

# API Configuration
API_HOST=0.0.0.0
API_PORT=8000

# Logging
LOG_LEVEL=INFO

# Future: Add real LLM configuration here

Usage

Start the server

uvicorn src.main:app --reload

API available at http://localhost:8000

API Endpoints

Responses API (Primary)

OpenAI Responses API format with structured output:

curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lorem-tester",
    "input": [
      {"role": "user", "content": "Explain quantum computing"}
    ],
    "reasoning": {
      "effort": "medium",
      "summary": "auto"
    },
    "max_output_tokens": 500,
    "stop": ["END"],
    "stream": false
  }'

Response Structure:

{
  "id": "resp_abc123",
  "object": "response",
  "created_at": 1733529600,
  "model": "lorem-tester",
  "status": "completed",
  "output": [
    {
      "type": "reasoning",
      "id": "reasoning_xyz",
      "summary": [
        "Analyzing the user's request...",
        "Considering quantum mechanics principles..."
      ]
    },
    {
      "type": "message",
      "id": "msg_def456",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Quantum computing uses quantum mechanics..."
        }
      ]
    }
  ],
  "usage": {
    "input_tokens": 10,
    "output_tokens": 50,
    "reasoning_tokens": 20,
    "total_tokens": 80
  }
}

Chat Completions (Compatibility)

OpenAI-compatible format with <think> tags:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lorem-tester",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ],
    "temperature": 0.7,
    "stream": true
  }'

Note: Chat Completions automatically enables reasoning and converts it to <think> tags for Open WebUI compatibility.

List Models

curl http://localhost:8000/v1/models

Returns:

{
  "object": "list",
  "data": [
    {
      "id": "lorem-tester",
      "object": "model",
      "created": 1733529600,
      "owned_by": "tatlock"
    },
    {
      "id": "tatlock",
      "object": "model",
      "created": 1733529600,
      "owned_by": "tatlock"
    }
  ]
}

Conversation History

Optional conversation tracking via metadata:

curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lorem-tester",
    "input": [
      {"role": "user", "content": "Hello"}
    ],
    "metadata": {
      "conversation_id": "conv_abc123"
    }
  }'

Hybrid Approach:

• Client MUST send full conversation history in input array (OpenAI compatible)
• Server optionally tracks via metadata.conversation_id (for analytics, future vector memory)
• Auto-generates conversation ID from first message hash if not provided

Interactive Documentation

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Open WebUI Integration

Docker Networking

If running Open WebUI in Docker and API on host:

# Use Docker bridge gateway IP
http://172.17.0.1:8000/v1/chat/completions

Reasoning Display

The Chat Completions wrapper automatically:

1. Enables reasoning generation
2. Converts reasoning items to <think> tags
3. Streams thinking before the actual response

Open WebUI displays this as:

• Thought bubble showing reasoning steps
• Main response showing the actual answer

Testing Error Handling

Lorem-tester supports error triggers:

• "trigger_rate_limit" - Simulates rate limit error
• "trigger_context_overflow" - Simulates context length error

Development

Project Structure

Following FastAPI best practices with domain-based organization:

tatlock/
├── src/
│   ├── agents/                # Agent interface and implementations
│   │   ├── base.py            # Abstract AgentInterface
│   │   ├── lorem_tester.py    # Full-featured mock agent
│   │   ├── tatlock.py         # Placeholder for real agent
│   │   └── registry.py        # Model registry
│   ├── responses/             # Responses API domain (PRIMARY)
│   │   ├── router.py          # POST /v1/responses
│   │   ├── schemas.py         # Request/response models
│   │   ├── service.py         # Response generation logic
│   │   ├── streaming.py       # SSE streaming coordinator
│   │   ├── history.py         # Conversation history management
│   │   └── context.py         # Context window management
│   ├── chat/                  # Chat Completions domain (WRAPPER)
│   │   ├── router.py          # POST /v1/chat/completions
│   │   ├── schemas.py         # Chat request/response models
│   │   ├── service.py         # Wraps Responses API
│   │   └── constants.py       # Chat constants
│   ├── models/                # Models listing domain
│   │   ├── router.py          # GET /v1/models
│   │   ├── schemas.py         # Model schemas
│   │   └── service.py         # Model registry access
│   ├── core/                  # Shared utilities
│   │   ├── config.py          # Configuration (BaseSettings)
│   │   ├── models.py          # Custom Pydantic base
│   │   ├── exceptions.py      # Custom exceptions
│   │   └── router.py          # Health check endpoints
│   └── main.py                # Application factory
├── tests/                     # Comprehensive test suite
│   ├── agents/                # Agent tests
│   ├── responses/             # Responses API tests
│   ├── chat/                  # Chat completions tests
│   ├── models/                # Models API tests
│   └── core/                  # Core tests
├── requirements.txt           # Dependencies (pinned)
├── .env                       # Environment variables
├── AGENTS.md                  # Agent documentation
├── CLEANUP_TODO.md            # Architecture notes
└── README.md                  # This file

Testing

# Run all tests
pytest

# Run with coverage
pytest --cov=src --cov-report=term-missing

# Current coverage: 78.95% (75 tests passing)

Test Organization:

• Unit tests for all components
• Integration tests for API endpoints
• Streaming tests for SSE functionality
• Error handling tests
• Advanced features tests (stop sequences, max tokens, validation)

Code Style

• Async-first: All I/O operations use async/await
• Type hints: All functions fully typed
• Pydantic validation: All request/response validation
• Domain separation: Clear boundaries between components
• Single responsibility: Each module has one clear purpose

Security

Version Locking

Minor version locking (>=X.Y,<X.(Y+1)) for security:

• Allows patch updates
• Blocks potentially breaking minor updates
• All dependencies checked for CVEs (2025-12-06)

Best Practices

1. Never commit .env files
2. Use environment variables for sensitive config
3. Keep dependencies updated monthly
4. Validate all inputs with Pydantic
5. Use HTTPS in production
6. Implement rate limiting

Deployment

Production Server

# Multiple workers for production
uvicorn src.main:app --host 0.0.0.0 --port 8000 --workers 4

Considerations

• Use reverse proxy (nginx/caddy) for HTTPS
• Enable rate limiting (SlowAPI or similar)
• Set up monitoring and logging
• Configure resource limits
• Use process manager (systemd/supervisor)

Troubleshooting

Common Issues

Streaming not working:

• Verify SSE-Starlette is installed
• Check client supports Server-Sent Events
• Test with: pytest tests/responses/ -k streaming

Open WebUI can't connect:

• Use Docker bridge gateway IP: 172.17.0.1:8000
• Check firewall settings
• Verify server is running on 0.0.0.0

Tests failing:

• Install test dependencies: pip install -r requirements-dev.txt
• Activate virtual environment
• Run with verbose: pytest -v

Reasoning not showing:

• Ensure using Chat Completions endpoint (auto-enables reasoning)
• Or manually enable in Responses API: "reasoning": {"effort": "medium", "summary": "auto"}
• Check Open WebUI version supports <think> tags

Future Roadmap

Short-term

• Connect tatlock model to real PydanticAI agent
• Implement vector memory (Qdrant integration)
• Add authentication/API keys
• Rate limiting middleware

Long-term

• Multi-model support (OpenAI, Anthropic, etc.)
• Advanced conversation memory
• Tool/function calling integration
• Usage tracking and analytics

Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes with tests
4. Ensure tests pass: pytest
5. Submit pull request

Documentation

• AGENTS.md: Agent architecture and best practices
• CLEANUP_TODO.md: Architecture decisions and future considerations
• CHANGELOG.md: Version history
• OpenAI Responses API: https://platform.openai.com/docs/api-reference/responses
• FastAPI: https://fastapi.tiangolo.com/
• PydanticAI: https://ai.pydantic.dev/

License

[Add your license here]

---

Note: This is a testing/development API with mock responses. The architecture is production-ready and designed for easy integration with real LLM backends (PydanticAI, Ollama, OpenAI, etc.).