tatlock/README.md

# Tatlock - OpenAI-Compatible API with Responses API

A FastAPI-based service providing OpenAI-compatible API endpoints with full Responses API support, reasoning display, and streaming. Features a hybrid architecture with chat completions as a compatibility wrapper around the Responses API.

## Current Status

**✅ Production-ready testing API** with OpenAI Responses API format
**✅ Open WebUI integration** with reasoning bubbles (`<think>` tags)
**✅ Conversation history** with hybrid client/server approach
**🚧 PydanticAI integration** prepared for future real LLM connection

## Architecture Overview

### Hybrid API Design

```
┌─────────────────────────────────────────┐
│  Client (Open WebUI, etc.)              │
└────────┬────────────────────────────────┘
         │
         ├──────────────────────────────────┐
         │                                   │
         v                                   v
┌────────────────────┐          ┌──────────────────────┐
│ /v1/chat/          │ wrapper  │ /v1/responses        │
│ completions        ├─────────>│ (Primary API)        │
│                    │          │                      │
│ • OpenAI compat    │          │ • Reasoning items    │
│ • <think> tags     │          │ • Function calls     │
│ • Legacy support   │          │ • Message items      │
└────────────────────┘          └──────────┬───────────┘
                                           │
                                           v
                                ┌──────────────────────┐
                                │  Agent Interface     │
                                │                      │
                                │ • lorem-tester       │
                                │ • tatlock (future)   │
                                └──────────────────────┘
```

**Key Architectural Decisions:**
- **Single Source of Truth**: Responses API handles all generation logic
- **Chat Completions Wrapper**: Converts Responses output to Chat format with `<think>` tags
- **Agent Interface**: Clean abstraction for multiple models (mock and real)
- **Hybrid History**: Client sends full context, server optionally tracks conversations

## Features

### Core API
- ✅ **Responses API** (`/v1/responses`) - Primary endpoint with structured output
  - Reasoning items (thinking/extended thinking)
  - Function call items (tool execution)
  - Message items (assistant responses)
  - Streaming and non-streaming modes
- ✅ **Chat Completions API** (`/v1/chat/completions`) - Compatibility wrapper
  - Converts reasoning to `<think>` tags for Open WebUI
  - Maintains OpenAI-compatible format
  - Wraps Responses API (single source of truth)
- ✅ **Models API** (`/v1/models`) - Lists available models

### Advanced Features
- ✅ **Conversation History Management**
  - Hybrid approach: client maintains state, server tracks optionally
  - Auto-generated conversation IDs from first message hash
  - Configurable max turns (default: 20)
  - Placeholder for future vector memory (Qdrant)
- ✅ **Context Window Management**
  - Approximate token counting (~4 chars/token)
  - Context trimming to fit model limits
  - Token usage statistics
- ✅ **Parameter Validation**
  - Temperature: 0.0-2.0
  - Reasoning effort: none, minimal, low, medium, high, xhigh
  - Max output tokens enforcement
  - Stop sequences (up to 4)
- ✅ **Stop Sequence Detection**
  - Real-time detection during streaming
  - Stops generation immediately when encountered
- ✅ **Max Tokens Enforcement**
  - Real-time token counting during streaming
  - Stops when limit reached

### Testing Models
- ✅ **lorem-tester** - Full-featured mock agent
  - Realistic reasoning summaries
  - Random tool/function call generation
  - Error triggers for testing (rate_limit, context_overflow)
  - Temperature variation
- ✅ **tatlock** - Placeholder for real PydanticAI agent

### Open WebUI Integration
- ✅ **Reasoning Display** - Thinking bubbles shown separately from responses
- ✅ **Streaming Support** - Smooth word-by-word streaming
- ✅ **Error Handling** - Graceful error display
- ✅ **Model Selection** - Both models available in dropdown

## Components

- **FastAPI**: High-performance web framework
- **SSE-Starlette**: Server-Sent Events for streaming
- **Pydantic**: Type-safe request/response validation
- **Agent Interface**: Abstraction for multiple model backends
- **Conversation History**: Server-side tracking with hybrid approach
- **Context Window**: Token management and trimming

## Requirements

- Python 3.12+ (Python 3.12.11 recommended)
- No external dependencies for mock API
- (Future: Network access for PydanticAI integration)

## Installation

### 1. Clone the repository

```bash
git clone <repository-url>
cd tatlock
```

### 2. Create a virtual environment

```bash
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
```

### 3. Install dependencies

```bash
pip install -r requirements.txt
```

### 4. Configure environment (Optional)

Create a `.env` file for custom configuration:

```env
# API Configuration
API_HOST=0.0.0.0
API_PORT=8000

# Logging
LOG_LEVEL=INFO

# Future: Add real LLM configuration here
```

## Usage

### Start the server

```bash
uvicorn src.main:app --reload
```

API available at `http://localhost:8000`

### API Endpoints

#### Responses API (Primary)

OpenAI Responses API format with structured output:

```bash
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lorem-tester",
    "input": [
      {"role": "user", "content": "Explain quantum computing"}
    ],
    "reasoning": {
      "effort": "medium",
      "summary": "auto"
    },
    "max_output_tokens": 500,
    "stop": ["END"],
    "stream": false
  }'
```

**Response Structure:**
```json
{
  "id": "resp_abc123",
  "object": "response",
  "created_at": 1733529600,
  "model": "lorem-tester",
  "status": "completed",
  "output": [
    {
      "type": "reasoning",
      "id": "reasoning_xyz",
      "summary": [
        "Analyzing the user's request...",
        "Considering quantum mechanics principles..."
      ]
    },
    {
      "type": "message",
      "id": "msg_def456",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Quantum computing uses quantum mechanics..."
        }
      ]
    }
  ],
  "usage": {
    "input_tokens": 10,
    "output_tokens": 50,
    "reasoning_tokens": 20,
    "total_tokens": 80
  }
}
```

#### Chat Completions (Compatibility)

OpenAI-compatible format with `<think>` tags:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lorem-tester",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ],
    "temperature": 0.7,
    "stream": true
  }'
```

**Note**: Chat Completions automatically enables reasoning and converts it to `<think>` tags for Open WebUI compatibility.

#### List Models

```bash
curl http://localhost:8000/v1/models
```

Returns:
```json
{
  "object": "list",
  "data": [
    {
      "id": "lorem-tester",
      "object": "model",
      "created": 1733529600,
      "owned_by": "tatlock"
    },
    {
      "id": "tatlock",
      "object": "model",
      "created": 1733529600,
      "owned_by": "tatlock"
    }
  ]
}
```

### Conversation History

Optional conversation tracking via metadata:

```bash
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lorem-tester",
    "input": [
      {"role": "user", "content": "Hello"}
    ],
    "metadata": {
      "conversation_id": "conv_abc123"
    }
  }'
```

**Hybrid Approach:**
- Client MUST send full conversation history in `input` array (OpenAI compatible)
- Server optionally tracks via `metadata.conversation_id` (for analytics, future vector memory)
- Auto-generates conversation ID from first message hash if not provided

### Interactive Documentation

- **Swagger UI**: `http://localhost:8000/docs`
- **ReDoc**: `http://localhost:8000/redoc`

## Open WebUI Integration

### Docker Networking

If running Open WebUI in Docker and API on host:

```bash
# Use Docker bridge gateway IP
http://172.17.0.1:8000/v1/chat/completions
```

### Reasoning Display

The Chat Completions wrapper automatically:
1. Enables reasoning generation
2. Converts reasoning items to `<think>` tags
3. Streams thinking before the actual response

Open WebUI displays this as:
- **Thought bubble** showing reasoning steps
- **Main response** showing the actual answer

### Testing Error Handling

Lorem-tester supports error triggers:
- **"trigger_rate_limit"** - Simulates rate limit error
- **"trigger_context_overflow"** - Simulates context length error

## Development

### Project Structure

Following FastAPI best practices with domain-based organization:

```
tatlock/
├── src/
│   ├── agents/                # Agent interface and implementations
│   │   ├── base.py            # Abstract AgentInterface
│   │   ├── lorem_tester.py    # Full-featured mock agent
│   │   ├── tatlock.py         # Placeholder for real agent
│   │   └── registry.py        # Model registry
│   ├── responses/             # Responses API domain (PRIMARY)
│   │   ├── router.py          # POST /v1/responses
│   │   ├── schemas.py         # Request/response models
│   │   ├── service.py         # Response generation logic
│   │   ├── streaming.py       # SSE streaming coordinator
│   │   ├── history.py         # Conversation history management
│   │   └── context.py         # Context window management
│   ├── chat/                  # Chat Completions domain (WRAPPER)
│   │   ├── router.py          # POST /v1/chat/completions
│   │   ├── schemas.py         # Chat request/response models
│   │   ├── service.py         # Wraps Responses API
│   │   └── constants.py       # Chat constants
│   ├── models/                # Models listing domain
│   │   ├── router.py          # GET /v1/models
│   │   ├── schemas.py         # Model schemas
│   │   └── service.py         # Model registry access
│   ├── core/                  # Shared utilities
│   │   ├── config.py          # Configuration (BaseSettings)
│   │   ├── models.py          # Custom Pydantic base
│   │   ├── exceptions.py      # Custom exceptions
│   │   └── router.py          # Health check endpoints
│   └── main.py                # Application factory
├── tests/                     # Comprehensive test suite
│   ├── agents/                # Agent tests
│   ├── responses/             # Responses API tests
│   ├── chat/                  # Chat completions tests
│   ├── models/                # Models API tests
│   └── core/                  # Core tests
├── requirements.txt           # Dependencies (pinned)
├── .env                       # Environment variables
├── AGENTS.md                  # Agent documentation
├── CLEANUP_TODO.md            # Architecture notes
└── README.md                  # This file
```

### Testing

```bash
# Run all tests
pytest

# Run with coverage
pytest --cov=src --cov-report=term-missing

# Current coverage: 78.95% (75 tests passing)
```

**Test Organization:**
- Unit tests for all components
- Integration tests for API endpoints
- Streaming tests for SSE functionality
- Error handling tests
- Advanced features tests (stop sequences, max tokens, validation)

### Code Style

- **Async-first**: All I/O operations use async/await
- **Type hints**: All functions fully typed
- **Pydantic validation**: All request/response validation
- **Domain separation**: Clear boundaries between components
- **Single responsibility**: Each module has one clear purpose

## Security

### Version Locking

Minor version locking (`>=X.Y,<X.(Y+1)`) for security:
- Allows patch updates
- Blocks potentially breaking minor updates
- All dependencies checked for CVEs (2025-12-06)

### Best Practices

1. Never commit `.env` files
2. Use environment variables for sensitive config
3. Keep dependencies updated monthly
4. Validate all inputs with Pydantic
5. Use HTTPS in production
6. Implement rate limiting

## Deployment

### Production Server

```bash
# Multiple workers for production
uvicorn src.main:app --host 0.0.0.0 --port 8000 --workers 4
```

### Considerations

- Use reverse proxy (nginx/caddy) for HTTPS
- Enable rate limiting (SlowAPI or similar)
- Set up monitoring and logging
- Configure resource limits
- Use process manager (systemd/supervisor)

## Troubleshooting

### Common Issues

**Streaming not working:**
- Verify SSE-Starlette is installed
- Check client supports Server-Sent Events
- Test with: `pytest tests/responses/ -k streaming`

**Open WebUI can't connect:**
- Use Docker bridge gateway IP: `172.17.0.1:8000`
- Check firewall settings
- Verify server is running on `0.0.0.0`

**Tests failing:**
- Install test dependencies: `pip install -r requirements-dev.txt`
- Activate virtual environment
- Run with verbose: `pytest -v`

**Reasoning not showing:**
- Ensure using Chat Completions endpoint (auto-enables reasoning)
- Or manually enable in Responses API: `"reasoning": {"effort": "medium", "summary": "auto"}`
- Check Open WebUI version supports `<think>` tags

## Future Roadmap

### Short-term
- [ ] Connect tatlock model to real PydanticAI agent
- [ ] Implement vector memory (Qdrant integration)
- [ ] Add authentication/API keys
- [ ] Rate limiting middleware

### Long-term
- [ ] Multi-model support (OpenAI, Anthropic, etc.)
- [ ] Advanced conversation memory
- [ ] Tool/function calling integration
- [ ] Usage tracking and analytics

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes with tests
4. Ensure tests pass: `pytest`
5. Submit pull request

## Documentation

- **AGENTS.md**: Agent architecture and best practices
- **CLEANUP_TODO.md**: Architecture decisions and future considerations
- **CHANGELOG.md**: Version history
- OpenAI Responses API: https://platform.openai.com/docs/api-reference/responses
- FastAPI: https://fastapi.tiangolo.com/
- PydanticAI: https://ai.pydantic.dev/

## License

[Add your license here]

---

**Note**: This is a testing/development API with mock responses. The architecture is production-ready and designed for easy integration with real LLM backends (PydanticAI, Ollama, OpenAI, etc.).