Update documentation with hybrid architecture
README.md: - Complete rewrite with hybrid architecture documentation - Architecture diagram showing wrapper pattern - Detailed feature list for all implemented phases - API usage examples for Responses and Chat Completions - Conversation history usage guide - Open WebUI integration instructions - Comprehensive troubleshooting section - Updated project structure - Deployment considerations AGENTS.md: - Current architecture section (as of 2025-12-06) - Hybrid architecture explanation - Key architectural decisions documented - Agent interface design patterns - Conversation history approach - Context window management - Testing infrastructure details - Implementation status updates - Coverage statistics (78.95%, 95 tests) Key Documentation Themes: - Single source of truth: Responses API - Wrapper pattern for Chat Completions - Hybrid conversation history approach - Clean agent abstraction - Production-ready testing infrastructure 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Claude
@@ -6,16 +6,67 @@ This document contains instructions and documentation references for AI assistan
|
|||||||
|
|
||||||
This project implements an OpenAI-compatible API endpoint using FastAPI, with streaming support. Currently returns mock responses - infrastructure prepared for future Ollama/PydanticAI integration.
|
This project implements an OpenAI-compatible API endpoint using FastAPI, with streaming support. Currently returns mock responses - infrastructure prepared for future Ollama/PydanticAI integration.
|
||||||
|
|
||||||
**Current State**: Production-ready mock API with OpenAI-compatible format
|
**Current State**: Production-ready testing API with Responses API and Open WebUI integration
|
||||||
**Future Integration**: Ollama and PydanticAI (client code ready, not connected)
|
**Future Integration**: PydanticAI for real LLM agents (tatlock model placeholder ready)
|
||||||
|
|
||||||
|
### Current Architecture (As of 2025-12-06)
|
||||||
|
|
||||||
|
This project implements a **hybrid architecture** with the Responses API as the primary endpoint and Chat Completions as a compatibility wrapper:
|
||||||
|
|
||||||
|
```
|
||||||
|
Client (Open WebUI)
|
||||||
|
↓
|
||||||
|
Chat Completions (/v1/chat/completions) → Wrapper
|
||||||
|
↓
|
||||||
|
Responses API (/v1/responses) → Primary
|
||||||
|
↓
|
||||||
|
Agent Interface (lorem-tester, tatlock)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Architectural Decisions:**
|
||||||
|
|
||||||
|
1. **Single Source of Truth**: All response generation happens in the Responses API
|
||||||
|
- Structured output with reasoning, function_call, and message items
|
||||||
|
- Real-time stop sequence and max tokens enforcement
|
||||||
|
- Conversation history tracking
|
||||||
|
- Context window management
|
||||||
|
|
||||||
|
2. **Chat Completions Wrapper**: Provides compatibility without duplicating logic
|
||||||
|
- Calls Responses API internally
|
||||||
|
- Automatically enables reasoning generation
|
||||||
|
- Converts reasoning items to `<think>` tags for Open WebUI
|
||||||
|
- Maintains OpenAI-compatible format
|
||||||
|
|
||||||
|
3. **Agent Interface**: Clean abstraction for multiple models
|
||||||
|
- **lorem-tester**: Full-featured mock agent with realistic behavior
|
||||||
|
- Reasoning summaries (adjustable effort levels)
|
||||||
|
- Random tool/function calls
|
||||||
|
- Error triggers for testing
|
||||||
|
- Temperature variation
|
||||||
|
- **tatlock**: Placeholder for future PydanticAI agent
|
||||||
|
|
||||||
|
4. **Hybrid Conversation History**:
|
||||||
|
- Client MUST send full context in `input` array (OpenAI compatible)
|
||||||
|
- Server optionally tracks via `metadata.conversation_id`
|
||||||
|
- Auto-generates deterministic IDs from first message
|
||||||
|
- Supports future vector memory integration (Qdrant)
|
||||||
|
|
||||||
|
**Why This Architecture?**
|
||||||
|
|
||||||
|
- **Open WebUI Compatibility**: Native Responses API support not yet in stable release
|
||||||
|
- **Future-Proof**: Easy migration when Open WebUI adds native support
|
||||||
|
- **Testability**: Full-featured mock agent (lorem-tester) for integration testing
|
||||||
|
- **Clean Separation**: Responses API as stable core, wrappers can change
|
||||||
|
|
||||||
### Components
|
### Components
|
||||||
|
|
||||||
- **FastAPI**: Web framework for the API layer
|
- **FastAPI**: Web framework for the API layer
|
||||||
- **SSE-Starlette**: Server-Sent Events for streaming responses
|
- **SSE-Starlette**: Server-Sent Events for streaming responses
|
||||||
- **Pydantic**: Request/response validation
|
- **Pydantic**: Request/response validation with field validators
|
||||||
- **PydanticAI**: Dependency installed, ready for future LLM integration
|
- **Agent Interface**: Abstract base class for model implementations
|
||||||
- **Ollama**: Async client implemented, ready for future connection
|
- **Conversation History**: Server-side tracking with configurable max turns
|
||||||
|
- **Context Window**: Token counting and management
|
||||||
|
- **PydanticAI**: Dependency installed, ready for tatlock agent implementation
|
||||||
|
|
||||||
## Documentation References
|
## Documentation References
|
||||||
|
|
||||||
@@ -104,17 +155,56 @@ This project implements an OpenAI-compatible API endpoint using FastAPI, with st
|
|||||||
#### OpenAI API Reference
|
#### OpenAI API Reference
|
||||||
- **Official Documentation**: https://platform.openai.com/docs/api-reference
|
- **Official Documentation**: https://platform.openai.com/docs/api-reference
|
||||||
- **Implemented Endpoints**:
|
- **Implemented Endpoints**:
|
||||||
- ✅ `/v1/chat/completions` - Chat completion with streaming (mock responses)
|
- ✅ `/v1/responses` - **Responses API (PRIMARY)** with structured output
|
||||||
- ✅ `/v1/models` - List available models (mock listing)
|
- Reasoning items (thinking summaries)
|
||||||
|
- Function call items (tool execution)
|
||||||
|
- Message items (assistant responses)
|
||||||
|
- Full streaming support with SSE
|
||||||
|
- Stop sequence detection
|
||||||
|
- Max tokens enforcement
|
||||||
|
- Conversation history tracking
|
||||||
|
- ✅ `/v1/chat/completions` - **Compatibility wrapper** around Responses API
|
||||||
|
- Converts reasoning to `<think>` tags for Open WebUI
|
||||||
|
- Automatically enables reasoning generation
|
||||||
|
- Maintains OpenAI-compatible format
|
||||||
|
- Supports streaming and non-streaming
|
||||||
|
- ✅ `/v1/models` - List available models (lorem-tester, tatlock)
|
||||||
- **Future Endpoints**:
|
- **Future Endpoints**:
|
||||||
- 🚧 `/v1/completions` - Text completion (legacy)
|
- 🚧 `/v1/completions` - Text completion (legacy)
|
||||||
- 🚧 `/v1/embeddings` - Text embeddings
|
- 🚧 `/v1/embeddings` - Text embeddings
|
||||||
- **Implemented Features**:
|
- **Implemented Features**:
|
||||||
- ✅ Streaming with Server-Sent Events
|
- ✅ **Responses API Format**:
|
||||||
- ✅ Message format compatibility
|
- Structured output items (reasoning, function_call, message)
|
||||||
- ✅ Response structure compatibility
|
- Extended thinking support
|
||||||
- ✅ OpenAI error format
|
- Tool/function calling support
|
||||||
- ✅ Request validation with Pydantic
|
- Streaming with multiple event types
|
||||||
|
- ✅ **Advanced Parameter Validation**:
|
||||||
|
- Temperature: 0.0-2.0 with Pydantic validators
|
||||||
|
- Reasoning effort: none, minimal, low, medium, high, xhigh
|
||||||
|
- Max output tokens: positive integer enforcement
|
||||||
|
- Stop sequences: up to 4, non-empty strings
|
||||||
|
- ✅ **Conversation History**:
|
||||||
|
- Hybrid client/server approach
|
||||||
|
- Auto-generated conversation IDs
|
||||||
|
- Configurable max turns (default: 20)
|
||||||
|
- Placeholder for vector memory
|
||||||
|
- ✅ **Context Management**:
|
||||||
|
- Approximate token counting (~4 chars/token)
|
||||||
|
- Context window trimming
|
||||||
|
- Usage statistics
|
||||||
|
- ✅ **Streaming Enforcement**:
|
||||||
|
- Real-time stop sequence detection
|
||||||
|
- Real-time max tokens enforcement
|
||||||
|
- Word-by-word streaming with delays
|
||||||
|
- ✅ **Error Handling**:
|
||||||
|
- Custom exception types (RateLimitError, ContextLengthError)
|
||||||
|
- OpenAI-compatible error format
|
||||||
|
- Error triggers in lorem-tester for testing
|
||||||
|
- ✅ **Testing Infrastructure**:
|
||||||
|
- 75 tests (78.95% coverage)
|
||||||
|
- Unit tests for all components
|
||||||
|
- Integration tests for API endpoints
|
||||||
|
- Streaming tests for SSE functionality
|
||||||
|
|
||||||
## FastAPI Best Practices
|
## FastAPI Best Practices
|
||||||
|
|
||||||
@@ -126,37 +216,49 @@ This project follows best practices from [github.com/zhanymkanov/fastapi-best-pr
|
|||||||
|
|
||||||
```
|
```
|
||||||
src/
|
src/
|
||||||
├── chat/ # Chat completions domain
|
├── agents/ # Agent interface and implementations
|
||||||
│ ├── router.py # FastAPI routes
|
│ ├── base.py # Abstract AgentInterface
|
||||||
│ ├── schemas.py # Pydantic request/response models
|
│ ├── lorem_tester.py # Full-featured mock agent
|
||||||
│ ├── service.py # Business logic
|
│ ├── tatlock.py # Placeholder for real agent
|
||||||
│ ├── dependencies.py # Domain-specific dependencies
|
│ └── registry.py # ModelRegistry for agent management
|
||||||
│ ├── constants.py # Domain constants
|
├── responses/ # Responses API domain (PRIMARY)
|
||||||
|
│ ├── router.py # POST /v1/responses endpoint
|
||||||
|
│ ├── schemas.py # Request/response models with validators
|
||||||
|
│ ├── service.py # Response generation logic
|
||||||
|
│ ├── streaming.py # SSE streaming coordinator
|
||||||
|
│ ├── history.py # Conversation history management
|
||||||
|
│ └── context.py # Context window and token management
|
||||||
|
├── chat/ # Chat Completions domain (WRAPPER)
|
||||||
|
│ ├── router.py # POST /v1/chat/completions endpoint
|
||||||
|
│ ├── schemas.py # Chat request/response models
|
||||||
|
│ ├── service.py # Wraps Responses API, converts to <think> tags
|
||||||
|
│ ├── constants.py # Chat constants (roles, finish reasons)
|
||||||
│ └── __init__.py
|
│ └── __init__.py
|
||||||
├── models/ # Models listing domain
|
├── models/ # Models listing domain
|
||||||
│ ├── router.py
|
│ ├── router.py # GET /v1/models endpoint
|
||||||
│ ├── schemas.py
|
│ ├── schemas.py # Model schemas
|
||||||
│ ├── service.py
|
│ ├── service.py # Accesses ModelRegistry
|
||||||
│ └── __init__.py
|
│ └── __init__.py
|
||||||
├── core/ # Shared utilities
|
├── core/ # Shared utilities
|
||||||
│ ├── config.py # Global configuration
|
│ ├── config.py # Global configuration (BaseSettings)
|
||||||
│ ├── models.py # Custom base Pydantic models
|
│ ├── models.py # Custom base Pydantic models
|
||||||
│ ├── exceptions.py # Global exceptions
|
│ ├── exceptions.py # Custom exceptions (RateLimitError, etc.)
|
||||||
│ ├── dependencies.py # Shared dependencies
|
│ ├── dependencies.py # Shared dependencies
|
||||||
│ └── router.py # Core routes (health, root)
|
│ └── router.py # Core routes (health, root)
|
||||||
├── ollama/ # Ollama client layer
|
├── ollama/ # Ollama client layer (not yet integrated)
|
||||||
│ ├── client.py # Async Ollama HTTP client
|
│ ├── client.py # Async Ollama HTTP client
|
||||||
│ ├── schemas.py # Ollama API models
|
│ └── schemas.py # Ollama API models
|
||||||
│ └── __init__.py
|
|
||||||
└── main.py # Application factory & configuration
|
└── main.py # Application factory & configuration
|
||||||
```
|
```
|
||||||
|
|
||||||
**Key Principles**:
|
**Key Architectural Principles**:
|
||||||
- Each domain has its own router, schemas, models, service, etc.
|
- **Single Source of Truth**: Responses API handles all generation logic
|
||||||
- Cross-domain imports use explicit naming: `from src.auth import constants as auth_constants`
|
- **Wrapper Pattern**: Chat Completions wraps Responses API without duplicating code
|
||||||
- Main.py focuses on configuration, middleware, and exception handlers
|
- **Agent Abstraction**: AgentInterface defines contract for all models
|
||||||
- Business logic stays in service modules
|
- **Domain Separation**: Each domain has its own router, schemas, service
|
||||||
- Routes delegate to services for all business logic
|
- **Service Layer**: Business logic in services, not routers
|
||||||
|
- **Type Safety**: Pydantic models for ALL request/response validation
|
||||||
|
- **Async First**: All I/O operations use async/await
|
||||||
|
|
||||||
### Async/Await Best Practices
|
### Async/Await Best Practices
|
||||||
|
|
||||||
|
|||||||
@@ -1,36 +1,114 @@
|
|||||||
# OpenAI-Compatible API
|
# Tatlock - OpenAI-Compatible API with Responses API
|
||||||
|
|
||||||
A FastAPI-based service that provides OpenAI-compatible API endpoints with streaming support. Currently returns mock responses - ready for future Ollama/PydanticAI integration.
|
A FastAPI-based service providing OpenAI-compatible API endpoints with full Responses API support, reasoning display, and streaming. Features a hybrid architecture with chat completions as a compatibility wrapper around the Responses API.
|
||||||
|
|
||||||
## Current Status
|
## Current Status
|
||||||
|
|
||||||
**✅ Production-ready mock API** with OpenAI-compatible format
|
**✅ Production-ready testing API** with OpenAI Responses API format
|
||||||
**🚧 Ollama/PydanticAI integration** prepared but not connected
|
**✅ Open WebUI integration** with reasoning bubbles (`<think>` tags)
|
||||||
|
**✅ Conversation history** with hybrid client/server approach
|
||||||
|
**🚧 PydanticAI integration** prepared for future real LLM connection
|
||||||
|
|
||||||
## Components
|
## Architecture Overview
|
||||||
|
|
||||||
- **FastAPI**: High-performance web framework providing the API layer
|
### Hybrid API Design
|
||||||
- **SSE-Starlette**: Server-Sent Events for streaming responses
|
|
||||||
- **Pydantic**: Type-safe request/response handling and validation
|
```
|
||||||
- **Ollama Client**: Async HTTP client prepared for future integration
|
┌─────────────────────────────────────────┐
|
||||||
- **PydanticAI**: Ready for LLM integration (not yet connected)
|
│ Client (Open WebUI, etc.) │
|
||||||
|
└────────┬────────────────────────────────┘
|
||||||
|
│
|
||||||
|
├──────────────────────────────────┐
|
||||||
|
│ │
|
||||||
|
v v
|
||||||
|
┌────────────────────┐ ┌──────────────────────┐
|
||||||
|
│ /v1/chat/ │ wrapper │ /v1/responses │
|
||||||
|
│ completions ├─────────>│ (Primary API) │
|
||||||
|
│ │ │ │
|
||||||
|
│ • OpenAI compat │ │ • Reasoning items │
|
||||||
|
│ • <think> tags │ │ • Function calls │
|
||||||
|
│ • Legacy support │ │ • Message items │
|
||||||
|
└────────────────────┘ └──────────┬───────────┘
|
||||||
|
│
|
||||||
|
v
|
||||||
|
┌──────────────────────┐
|
||||||
|
│ Agent Interface │
|
||||||
|
│ │
|
||||||
|
│ • lorem-tester │
|
||||||
|
│ • tatlock (future) │
|
||||||
|
└──────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Architectural Decisions:**
|
||||||
|
- **Single Source of Truth**: Responses API handles all generation logic
|
||||||
|
- **Chat Completions Wrapper**: Converts Responses output to Chat format with `<think>` tags
|
||||||
|
- **Agent Interface**: Clean abstraction for multiple models (mock and real)
|
||||||
|
- **Hybrid History**: Client sends full context, server optionally tracks conversations
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
- ✅ OpenAI-compatible API endpoints (`/v1/chat/completions`, `/v1/models`)
|
### Core API
|
||||||
- ✅ Streaming responses with Server-Sent Events (SSE)
|
- ✅ **Responses API** (`/v1/responses`) - Primary endpoint with structured output
|
||||||
- ✅ Type-safe request/response handling with Pydantic
|
- Reasoning items (thinking/extended thinking)
|
||||||
- ✅ Async/await throughout for optimal performance
|
- Function call items (tool execution)
|
||||||
- ✅ Comprehensive test suite (62% coverage)
|
- Message items (assistant responses)
|
||||||
- ✅ Domain-based architecture following FastAPI best practices
|
- Streaming and non-streaming modes
|
||||||
- 🚧 Ollama integration (client ready, not connected)
|
- ✅ **Chat Completions API** (`/v1/chat/completions`) - Compatibility wrapper
|
||||||
- 🚧 PydanticAI integration (dependency installed, not connected)
|
- Converts reasoning to `<think>` tags for Open WebUI
|
||||||
|
- Maintains OpenAI-compatible format
|
||||||
|
- Wraps Responses API (single source of truth)
|
||||||
|
- ✅ **Models API** (`/v1/models`) - Lists available models
|
||||||
|
|
||||||
|
### Advanced Features
|
||||||
|
- ✅ **Conversation History Management**
|
||||||
|
- Hybrid approach: client maintains state, server tracks optionally
|
||||||
|
- Auto-generated conversation IDs from first message hash
|
||||||
|
- Configurable max turns (default: 20)
|
||||||
|
- Placeholder for future vector memory (Qdrant)
|
||||||
|
- ✅ **Context Window Management**
|
||||||
|
- Approximate token counting (~4 chars/token)
|
||||||
|
- Context trimming to fit model limits
|
||||||
|
- Token usage statistics
|
||||||
|
- ✅ **Parameter Validation**
|
||||||
|
- Temperature: 0.0-2.0
|
||||||
|
- Reasoning effort: none, minimal, low, medium, high, xhigh
|
||||||
|
- Max output tokens enforcement
|
||||||
|
- Stop sequences (up to 4)
|
||||||
|
- ✅ **Stop Sequence Detection**
|
||||||
|
- Real-time detection during streaming
|
||||||
|
- Stops generation immediately when encountered
|
||||||
|
- ✅ **Max Tokens Enforcement**
|
||||||
|
- Real-time token counting during streaming
|
||||||
|
- Stops when limit reached
|
||||||
|
|
||||||
|
### Testing Models
|
||||||
|
- ✅ **lorem-tester** - Full-featured mock agent
|
||||||
|
- Realistic reasoning summaries
|
||||||
|
- Random tool/function call generation
|
||||||
|
- Error triggers for testing (rate_limit, context_overflow)
|
||||||
|
- Temperature variation
|
||||||
|
- ✅ **tatlock** - Placeholder for real PydanticAI agent
|
||||||
|
|
||||||
|
### Open WebUI Integration
|
||||||
|
- ✅ **Reasoning Display** - Thinking bubbles shown separately from responses
|
||||||
|
- ✅ **Streaming Support** - Smooth word-by-word streaming
|
||||||
|
- ✅ **Error Handling** - Graceful error display
|
||||||
|
- ✅ **Model Selection** - Both models available in dropdown
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
- **FastAPI**: High-performance web framework
|
||||||
|
- **SSE-Starlette**: Server-Sent Events for streaming
|
||||||
|
- **Pydantic**: Type-safe request/response validation
|
||||||
|
- **Agent Interface**: Abstraction for multiple model backends
|
||||||
|
- **Conversation History**: Server-side tracking with hybrid approach
|
||||||
|
- **Context Window**: Token management and trimming
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
- Python 3.12+ (Python 3.12.11 recommended for security)
|
- Python 3.12+ (Python 3.12.11 recommended)
|
||||||
- No external dependencies required for mock API
|
- No external dependencies for mock API
|
||||||
- (Future: Network access to Ollama instance for LLM integration)
|
- (Future: Network access for PydanticAI integration)
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
@@ -44,8 +122,8 @@ cd tatlock
|
|||||||
### 2. Create a virtual environment
|
### 2. Create a virtual environment
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -m venv venv
|
python -m venv .venv
|
||||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Install dependencies
|
### 3. Install dependencies
|
||||||
@@ -54,9 +132,9 @@ source venv/bin/activate # On Windows: venv\Scripts\activate
|
|||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4. Configure environment variables (Optional)
|
### 4. Configure environment (Optional)
|
||||||
|
|
||||||
Create a `.env` file in the project root for custom configuration:
|
Create a `.env` file for custom configuration:
|
||||||
|
|
||||||
```env
|
```env
|
||||||
# API Configuration
|
# API Configuration
|
||||||
@@ -66,231 +144,348 @@ API_PORT=8000
|
|||||||
# Logging
|
# Logging
|
||||||
LOG_LEVEL=INFO
|
LOG_LEVEL=INFO
|
||||||
|
|
||||||
# Future Ollama Configuration (not yet integrated)
|
# Future: Add real LLM configuration here
|
||||||
# OLLAMA_HOST=http://localhost:11434
|
|
||||||
# OLLAMA_DEFAULT_MODEL=mistral-nemo:latest
|
|
||||||
# OLLAMA_TIMEOUT=120
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Note**: The API works with defaults. Environment variables are optional for customization. Ollama configuration is prepared but not currently used.
|
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
### Start the development server
|
### Start the server
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uvicorn src.main:app --reload
|
uvicorn src.main:app --reload
|
||||||
```
|
```
|
||||||
|
|
||||||
The API will be available at `http://localhost:8000`
|
API available at `http://localhost:8000`
|
||||||
|
|
||||||
### API Endpoints
|
### API Endpoints
|
||||||
|
|
||||||
#### Chat Completions (OpenAI-compatible)
|
#### Responses API (Primary)
|
||||||
|
|
||||||
Returns mock lorem ipsum responses:
|
OpenAI Responses API format with structured output:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8000/v1/responses \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "lorem-tester",
|
||||||
|
"input": [
|
||||||
|
{"role": "user", "content": "Explain quantum computing"}
|
||||||
|
],
|
||||||
|
"reasoning": {
|
||||||
|
"effort": "medium",
|
||||||
|
"summary": "auto"
|
||||||
|
},
|
||||||
|
"max_output_tokens": 500,
|
||||||
|
"stop": ["END"],
|
||||||
|
"stream": false
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response Structure:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "resp_abc123",
|
||||||
|
"object": "response",
|
||||||
|
"created_at": 1733529600,
|
||||||
|
"model": "lorem-tester",
|
||||||
|
"status": "completed",
|
||||||
|
"output": [
|
||||||
|
{
|
||||||
|
"type": "reasoning",
|
||||||
|
"id": "reasoning_xyz",
|
||||||
|
"summary": [
|
||||||
|
"Analyzing the user's request...",
|
||||||
|
"Considering quantum mechanics principles..."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "message",
|
||||||
|
"id": "msg_def456",
|
||||||
|
"role": "assistant",
|
||||||
|
"content": [
|
||||||
|
{
|
||||||
|
"type": "output_text",
|
||||||
|
"text": "Quantum computing uses quantum mechanics..."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"usage": {
|
||||||
|
"input_tokens": 10,
|
||||||
|
"output_tokens": 50,
|
||||||
|
"reasoning_tokens": 20,
|
||||||
|
"total_tokens": 80
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Chat Completions (Compatibility)
|
||||||
|
|
||||||
|
OpenAI-compatible format with `<think>` tags:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl http://localhost:8000/v1/chat/completions \
|
curl http://localhost:8000/v1/chat/completions \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-d '{
|
-d '{
|
||||||
"model": "mistral-nemo:latest",
|
"model": "lorem-tester",
|
||||||
"messages": [
|
"messages": [
|
||||||
{"role": "user", "content": "Hello, how are you?"}
|
{"role": "user", "content": "Hello!"}
|
||||||
],
|
],
|
||||||
|
"temperature": 0.7,
|
||||||
"stream": true
|
"stream": true
|
||||||
}'
|
}'
|
||||||
```
|
```
|
||||||
|
|
||||||
#### List Models
|
**Note**: Chat Completions automatically enables reasoning and converts it to `<think>` tags for Open WebUI compatibility.
|
||||||
|
|
||||||
Returns mock model listing:
|
#### List Models
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl http://localhost:8000/v1/models
|
curl http://localhost:8000/v1/models
|
||||||
```
|
```
|
||||||
|
|
||||||
Response: `{"object": "list", "data": [{"id": "mistral-nemo:latest", ...}]}`
|
Returns:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"object": "list",
|
||||||
|
"data": [
|
||||||
|
{
|
||||||
|
"id": "lorem-tester",
|
||||||
|
"object": "model",
|
||||||
|
"created": 1733529600,
|
||||||
|
"owned_by": "tatlock"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "tatlock",
|
||||||
|
"object": "model",
|
||||||
|
"created": 1733529600,
|
||||||
|
"owned_by": "tatlock"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
### Interactive API Documentation
|
### Conversation History
|
||||||
|
|
||||||
- Swagger UI: `http://localhost:8000/docs`
|
Optional conversation tracking via metadata:
|
||||||
- ReDoc: `http://localhost:8000/redoc`
|
|
||||||
|
|
||||||
## Security
|
```bash
|
||||||
|
curl http://localhost:8000/v1/responses \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "lorem-tester",
|
||||||
|
"input": [
|
||||||
|
{"role": "user", "content": "Hello"}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"conversation_id": "conv_abc123"
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
### Version Locking Strategy
|
**Hybrid Approach:**
|
||||||
|
- Client MUST send full conversation history in `input` array (OpenAI compatible)
|
||||||
|
- Server optionally tracks via `metadata.conversation_id` (for analytics, future vector memory)
|
||||||
|
- Auto-generates conversation ID from first message hash if not provided
|
||||||
|
|
||||||
This project uses minor version locking (`>=X.Y,<X.(Y+1)`) to protect against supply chain attacks while allowing patch updates. All dependencies have been:
|
### Interactive Documentation
|
||||||
|
|
||||||
- Checked for known CVEs (as of 2025-12-06)
|
- **Swagger UI**: `http://localhost:8000/docs`
|
||||||
- Pinned to secure minor versions
|
- **ReDoc**: `http://localhost:8000/redoc`
|
||||||
- Documented with version rationale in `requirements.txt`
|
|
||||||
|
|
||||||
### CVE Status (2025-12-06)
|
## Open WebUI Integration
|
||||||
|
|
||||||
- **FastAPI 0.123.9**: No known vulnerabilities
|
### Docker Networking
|
||||||
- **Uvicorn 0.38.0**: No known vulnerabilities
|
|
||||||
- **PydanticAI 1.27.0**: No known vulnerabilities
|
|
||||||
- **HTTPX 0.28.1**: No known vulnerabilities
|
|
||||||
- **SSE-Starlette 3.0.2**: No known vulnerabilities
|
|
||||||
|
|
||||||
Regular security updates are recommended. Check for new versions monthly.
|
If running Open WebUI in Docker and API on host:
|
||||||
|
|
||||||
### Security Best Practices
|
```bash
|
||||||
|
# Use Docker bridge gateway IP
|
||||||
|
http://172.17.0.1:8000/v1/chat/completions
|
||||||
|
```
|
||||||
|
|
||||||
1. Never commit `.env` files
|
### Reasoning Display
|
||||||
2. Use environment variables for sensitive configuration
|
|
||||||
3. Keep dependencies updated
|
The Chat Completions wrapper automatically:
|
||||||
4. Implement rate limiting in production
|
1. Enables reasoning generation
|
||||||
5. Use HTTPS in production environments
|
2. Converts reasoning items to `<think>` tags
|
||||||
6. Validate all inputs with Pydantic models
|
3. Streams thinking before the actual response
|
||||||
|
|
||||||
|
Open WebUI displays this as:
|
||||||
|
- **Thought bubble** showing reasoning steps
|
||||||
|
- **Main response** showing the actual answer
|
||||||
|
|
||||||
|
### Testing Error Handling
|
||||||
|
|
||||||
|
Lorem-tester supports error triggers:
|
||||||
|
- **"trigger_rate_limit"** - Simulates rate limit error
|
||||||
|
- **"trigger_context_overflow"** - Simulates context length error
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
### Project Structure
|
### Project Structure
|
||||||
|
|
||||||
Following [FastAPI best practices](https://github.com/zhanymkanov/fastapi-best-practices) with domain-based organization:
|
Following FastAPI best practices with domain-based organization:
|
||||||
|
|
||||||
```
|
```
|
||||||
tatlock/
|
tatlock/
|
||||||
├── src/
|
├── src/
|
||||||
│ ├── chat/ # Chat completions domain
|
│ ├── agents/ # Agent interface and implementations
|
||||||
│ │ ├── router.py # OpenAI-compatible /v1/chat/completions
|
│ │ ├── base.py # Abstract AgentInterface
|
||||||
|
│ │ ├── lorem_tester.py # Full-featured mock agent
|
||||||
|
│ │ ├── tatlock.py # Placeholder for real agent
|
||||||
|
│ │ └── registry.py # Model registry
|
||||||
|
│ ├── responses/ # Responses API domain (PRIMARY)
|
||||||
|
│ │ ├── router.py # POST /v1/responses
|
||||||
│ │ ├── schemas.py # Request/response models
|
│ │ ├── schemas.py # Request/response models
|
||||||
│ │ ├── service.py # Business logic (currently mock)
|
│ │ ├── service.py # Response generation logic
|
||||||
│ │ ├── dependencies.py # Route dependencies
|
│ │ ├── streaming.py # SSE streaming coordinator
|
||||||
│ │ └── constants.py # Domain constants
|
│ │ ├── history.py # Conversation history management
|
||||||
|
│ │ └── context.py # Context window management
|
||||||
|
│ ├── chat/ # Chat Completions domain (WRAPPER)
|
||||||
|
│ │ ├── router.py # POST /v1/chat/completions
|
||||||
|
│ │ ├── schemas.py # Chat request/response models
|
||||||
|
│ │ ├── service.py # Wraps Responses API
|
||||||
|
│ │ └── constants.py # Chat constants
|
||||||
│ ├── models/ # Models listing domain
|
│ ├── models/ # Models listing domain
|
||||||
│ │ ├── router.py # OpenAI-compatible /v1/models
|
│ │ ├── router.py # GET /v1/models
|
||||||
│ │ ├── schemas.py # Model schemas
|
│ │ ├── schemas.py # Model schemas
|
||||||
│ │ └── service.py # Model list service (currently mock)
|
│ │ └── service.py # Model registry access
|
||||||
│ ├── core/ # Shared utilities
|
│ ├── core/ # Shared utilities
|
||||||
│ │ ├── config.py # Global configuration (BaseSettings)
|
│ │ ├── config.py # Configuration (BaseSettings)
|
||||||
│ │ ├── models.py # Custom Pydantic base models
|
│ │ ├── models.py # Custom Pydantic base
|
||||||
│ │ ├── exceptions.py # Custom exceptions
|
│ │ ├── exceptions.py # Custom exceptions
|
||||||
│ │ ├── router.py # Health check & root endpoints
|
│ │ └── router.py # Health check endpoints
|
||||||
│ │ └── dependencies.py # Shared dependencies
|
│ └── main.py # Application factory
|
||||||
│ ├── ollama/ # Ollama client (ready, not integrated yet)
|
├── tests/ # Comprehensive test suite
|
||||||
│ │ ├── client.py # Async HTTP client
|
│ ├── agents/ # Agent tests
|
||||||
│ │ └── schemas.py # Ollama API models
|
│ ├── responses/ # Responses API tests
|
||||||
│ └── main.py # Application factory & configuration
|
│ ├── chat/ # Chat completions tests
|
||||||
├── requirements.txt # Python dependencies (pinned)
|
│ ├── models/ # Models API tests
|
||||||
├── .env # Environment variables (git-ignored)
|
│ └── core/ # Core tests
|
||||||
├── .gitignore # Git ignore rules
|
├── requirements.txt # Dependencies (pinned)
|
||||||
├── AGENTS.md # LLM agent documentation + best practices
|
├── .env # Environment variables
|
||||||
└── README.md # This file
|
├── AGENTS.md # Agent documentation
|
||||||
|
├── CLEANUP_TODO.md # Architecture notes
|
||||||
|
└── README.md # This file
|
||||||
```
|
```
|
||||||
|
|
||||||
**Key Architectural Decisions**:
|
|
||||||
- **Domain-based** structure (not file-type based)
|
|
||||||
- **Separation of concerns**: Routers → Services → Clients
|
|
||||||
- **Factory pattern** in main.py for testability
|
|
||||||
- **Custom base models** for consistent serialization
|
|
||||||
- **Async-first** for all I/O operations
|
|
||||||
|
|
||||||
### Code Style
|
|
||||||
|
|
||||||
Following FastAPI best practices:
|
|
||||||
- **Async routes** for ALL I/O operations (HTTP, database, file access)
|
|
||||||
- **Sync routes** only for CPU-intensive work or blocking SDKs
|
|
||||||
- **Type hints** on all functions and class attributes
|
|
||||||
- **Pydantic models** for ALL request/response validation
|
|
||||||
- **Dependency injection** for validation and shared resources
|
|
||||||
- **Business logic** in service modules, NOT in routers
|
|
||||||
- Follow PEP 8 style guidelines
|
|
||||||
- Document complex logic with docstrings
|
|
||||||
|
|
||||||
### Current Implementation Status
|
|
||||||
|
|
||||||
The API is currently set up with **mock responses** for development:
|
|
||||||
|
|
||||||
**✅ Implemented**:
|
|
||||||
- OpenAI-compatible API structure
|
|
||||||
- `/v1/chat/completions` endpoint (returns lorem ipsum)
|
|
||||||
- `/v1/models` endpoint (returns mistral-nemo:latest)
|
|
||||||
- `/health` and `/` endpoints
|
|
||||||
- Streaming support with SSE
|
|
||||||
- Exception handling
|
|
||||||
- Configuration management
|
|
||||||
- Async Ollama client (ready, not connected)
|
|
||||||
|
|
||||||
**🚧 TODO** (future integration):
|
|
||||||
- Connect chat completions to Ollama/PydanticAI
|
|
||||||
- Implement actual model listing from Ollama
|
|
||||||
- Add authentication/API keys
|
|
||||||
- Rate limiting
|
|
||||||
- Usage tracking
|
|
||||||
- More OpenAI-compatible endpoints
|
|
||||||
|
|
||||||
### Testing
|
### Testing
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Install test dependencies
|
# Run all tests
|
||||||
pip install -r requirements-dev.txt
|
|
||||||
|
|
||||||
# Run tests
|
|
||||||
pytest
|
pytest
|
||||||
|
|
||||||
# Run tests with coverage
|
# Run with coverage
|
||||||
pytest --cov=src --cov-report=term-missing
|
pytest --cov=src --cov-report=term-missing
|
||||||
|
|
||||||
# Current coverage: 62%
|
# Current coverage: 78.95% (75 tests passing)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Documentation
|
**Test Organization:**
|
||||||
|
- Unit tests for all components
|
||||||
|
- Integration tests for API endpoints
|
||||||
|
- Streaming tests for SSE functionality
|
||||||
|
- Error handling tests
|
||||||
|
- Advanced features tests (stop sequences, max tokens, validation)
|
||||||
|
|
||||||
- See `AGENTS.md` for LLM agent instructions and package documentation
|
### Code Style
|
||||||
- FastAPI docs: https://fastapi.tiangolo.com/
|
|
||||||
- PydanticAI docs: https://ai.pydantic.dev/
|
- **Async-first**: All I/O operations use async/await
|
||||||
- Ollama API: https://github.com/ollama/ollama/blob/main/docs/api.md
|
- **Type hints**: All functions fully typed
|
||||||
|
- **Pydantic validation**: All request/response validation
|
||||||
|
- **Domain separation**: Clear boundaries between components
|
||||||
|
- **Single responsibility**: Each module has one clear purpose
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
### Version Locking
|
||||||
|
|
||||||
|
Minor version locking (`>=X.Y,<X.(Y+1)`) for security:
|
||||||
|
- Allows patch updates
|
||||||
|
- Blocks potentially breaking minor updates
|
||||||
|
- All dependencies checked for CVEs (2025-12-06)
|
||||||
|
|
||||||
|
### Best Practices
|
||||||
|
|
||||||
|
1. Never commit `.env` files
|
||||||
|
2. Use environment variables for sensitive config
|
||||||
|
3. Keep dependencies updated monthly
|
||||||
|
4. Validate all inputs with Pydantic
|
||||||
|
5. Use HTTPS in production
|
||||||
|
6. Implement rate limiting
|
||||||
|
|
||||||
## Deployment
|
## Deployment
|
||||||
|
|
||||||
### Docker Deployment (Coming Soon)
|
### Production Server
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker-compose up -d
|
# Multiple workers for production
|
||||||
|
uvicorn src.main:app --host 0.0.0.0 --port 8000 --workers 4
|
||||||
```
|
```
|
||||||
|
|
||||||
### Production Considerations
|
### Considerations
|
||||||
|
|
||||||
- Use a production ASGI server (uvicorn with multiple workers)
|
- Use reverse proxy (nginx/caddy) for HTTPS
|
||||||
- Enable HTTPS with reverse proxy (nginx/caddy)
|
- Enable rate limiting (SlowAPI or similar)
|
||||||
- Implement rate limiting
|
|
||||||
- Set up monitoring and logging
|
- Set up monitoring and logging
|
||||||
- Use a process manager (systemd/supervisor)
|
- Configure resource limits
|
||||||
- Configure proper resource limits
|
- Use process manager (systemd/supervisor)
|
||||||
- (Future) Ensure reliable network connectivity to Ollama instance
|
|
||||||
- (Future) Consider Ollama failover/redundancy strategies
|
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
### Common Issues
|
### Common Issues
|
||||||
|
|
||||||
**Issue**: Import errors
|
**Streaming not working:**
|
||||||
- Ensure virtual environment is activated
|
|
||||||
- Reinstall dependencies: `pip install -r requirements.txt`
|
|
||||||
- Verify Python 3.12+ is being used
|
|
||||||
|
|
||||||
**Issue**: Streaming not working
|
|
||||||
- Verify SSE-Starlette is installed
|
- Verify SSE-Starlette is installed
|
||||||
- Check client supports Server-Sent Events
|
- Check client supports Server-Sent Events
|
||||||
- Review browser/tool compatibility
|
- Test with: `pytest tests/responses/ -k streaming`
|
||||||
- Check test suite: `pytest tests/chat/test_router.py -k streaming`
|
|
||||||
|
|
||||||
**Issue**: Tests failing
|
**Open WebUI can't connect:**
|
||||||
|
- Use Docker bridge gateway IP: `172.17.0.1:8000`
|
||||||
|
- Check firewall settings
|
||||||
|
- Verify server is running on `0.0.0.0`
|
||||||
|
|
||||||
|
**Tests failing:**
|
||||||
- Install test dependencies: `pip install -r requirements-dev.txt`
|
- Install test dependencies: `pip install -r requirements-dev.txt`
|
||||||
- Check async test configuration in `pytest.ini`
|
- Activate virtual environment
|
||||||
- Run with verbose output: `pytest -v`
|
- Run with verbose: `pytest -v`
|
||||||
|
|
||||||
|
**Reasoning not showing:**
|
||||||
|
- Ensure using Chat Completions endpoint (auto-enables reasoning)
|
||||||
|
- Or manually enable in Responses API: `"reasoning": {"effort": "medium", "summary": "auto"}`
|
||||||
|
- Check Open WebUI version supports `<think>` tags
|
||||||
|
|
||||||
|
## Future Roadmap
|
||||||
|
|
||||||
|
### Short-term
|
||||||
|
- [ ] Connect tatlock model to real PydanticAI agent
|
||||||
|
- [ ] Implement vector memory (Qdrant integration)
|
||||||
|
- [ ] Add authentication/API keys
|
||||||
|
- [ ] Rate limiting middleware
|
||||||
|
|
||||||
|
### Long-term
|
||||||
|
- [ ] Multi-model support (OpenAI, Anthropic, etc.)
|
||||||
|
- [ ] Advanced conversation memory
|
||||||
|
- [ ] Tool/function calling integration
|
||||||
|
- [ ] Usage tracking and analytics
|
||||||
|
|
||||||
## Contributing
|
## Contributing
|
||||||
|
|
||||||
1. Fork the repository
|
1. Fork the repository
|
||||||
2. Create a feature branch
|
2. Create a feature branch
|
||||||
3. Make your changes
|
3. Make changes with tests
|
||||||
4. Add tests
|
4. Ensure tests pass: `pytest`
|
||||||
5. Submit a pull request
|
5. Submit pull request
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- **AGENTS.md**: Agent architecture and best practices
|
||||||
|
- **CLEANUP_TODO.md**: Architecture decisions and future considerations
|
||||||
|
- **CHANGELOG.md**: Version history
|
||||||
|
- OpenAI Responses API: https://platform.openai.com/docs/api-reference/responses
|
||||||
|
- FastAPI: https://fastapi.tiangolo.com/
|
||||||
|
- PydanticAI: https://ai.pydantic.dev/
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
@@ -298,5 +493,4 @@ docker-compose up -d
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
For detailed changelog, see [CHANGELOG.md](CHANGELOG.md).
|
**Note**: This is a testing/development API with mock responses. The architecture is production-ready and designed for easy integration with real LLM backends (PydanticAI, Ollama, OpenAI, etc.).
|
||||||
For AI assistant instructions and package documentation, see [AGENTS.md](AGENTS.md).
|
|
||||||
|
|||||||
Reference in New Issue
Block a user