From 0e810244bb2de9d9dfc6192ba6355d17877005f3 Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Sat, 6 Dec 2025 10:54:40 +0100 Subject: [PATCH] Add comprehensive project documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete documentation for setup, usage, and development. Includes LLM agent instructions and changelog. README.md: - Project overview and features - Requirements (Python 3.12.11, Ollama) - Installation instructions - Configuration guide (.env setup) - Running instructions (dev and production) - Testing guide (pytest, coverage) - API endpoint documentation - Project structure explanation - Development workflow - Security features - License information AGENTS.md: - LLM agent instructions - Project context and architecture - Domain-based structure details - Best practices documentation - FastAPI patterns and conventions - Testing strategies - Code style guidelines - Common tasks and operations - Ollama integration notes - Security considerations CHANGELOG.md: - Keep a Changelog format - Semantic versioning (v0.1.0) - Unreleased changes section - Detailed feature tracking - Security notes (CVE checks) - Version history with dates - GitHub release links Documentation Highlights: - Clear setup instructions - Environment configuration - Testing commands - Project structure - Security-focused - LLM-friendly instructions Following Standards: - Keep a Changelog format - Semantic versioning - Clear project structure - Comprehensive coverage Status: Production-ready documentation πŸ€– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- AGENTS.md | 352 +++++++++++++++++++++++++++++++++++++++++++++++++++ CHANGELOG.md | 45 +++++++ README.md | 304 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 701 insertions(+) create mode 100644 AGENTS.md create mode 100644 CHANGELOG.md create mode 100644 README.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..75f2bb7 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,352 @@ +# LLM Agent Instructions + +This document contains instructions and documentation references for AI assistants working with this codebase. + +## Project Overview + +This project implements an OpenAI-compatible API endpoint using FastAPI, with streaming support for LLM responses. The architecture consists of: + +- **FastAPI**: Web framework for the API layer +- **PydanticAI**: Agent framework for LLM integration +- **Ollama**: LLM backend running on a networked container +- **Stream Coordinator**: Manages streaming responses in OpenAI-compatible format + +## Documentation References + +### Core Framework Documentation + +#### FastAPI +- **Official Documentation**: https://fastapi.tiangolo.com/ +- **Version**: 0.123.9 (Dec 2025) +- **Key Topics**: + - Path operations and routing + - Request/response models with Pydantic + - Dependency injection + - Background tasks + - WebSocket and streaming support +- **PyPI**: https://pypi.org/project/fastapi/ + +#### Uvicorn +- **Official Documentation**: https://www.uvicorn.org/ +- **Version**: 0.38.0 (Oct 2025) +- **Key Topics**: + - ASGI server configuration + - Deployment settings + - Logging and monitoring + - SSL/TLS configuration + +### AI/LLM Integration + +#### PydanticAI +- **Official Documentation**: https://ai.pydantic.dev/ +- **Version**: 1.27.0 (Dec 2025) +- **Key Topics**: + - Agent creation and configuration + - LLM provider integration (Ollama support) + - Structured outputs with Pydantic + - Streaming responses + - Tool/function calling + - RunContext and dynamic configuration + - MCP server integration +- **GitHub**: https://github.com/pydantic/pydantic-ai +- **PyPI**: https://pypi.org/project/pydantic-ai/ + +#### Pydantic +- **Official Documentation**: https://docs.pydantic.dev/latest/ +- **Version**: 2.10+ (Required for PydanticAI) +- **Key Topics**: + - Data validation and serialization + - Field types and validators + - Model configuration + - JSON schema generation + +### HTTP and Streaming + +#### HTTPX +- **Official Documentation**: https://www.python-httpx.org/ +- **Version**: 0.28.1 +- **Key Topics**: + - Async HTTP client for Ollama communication + - Streaming responses + - Timeout configuration + - Connection pooling + +#### SSE-Starlette +- **GitHub**: https://github.com/sysid/sse-starlette +- **Version**: 3.0.2 (Oct 2025) +- **Key Topics**: + - Server-Sent Events implementation + - Streaming event responses + - Integration with FastAPI/Starlette + +### Ollama Integration + +#### Ollama API +- **Official Documentation**: https://github.com/ollama/ollama/blob/main/docs/api.md +- **Key Topics**: + - REST API endpoints + - Streaming responses + - Model management + - Generate and chat endpoints + - Model configuration + +### OpenAI API Compatibility + +#### OpenAI API Reference +- **Official Documentation**: https://platform.openai.com/docs/api-reference +- **Key Endpoints to Implement**: + - `/v1/chat/completions` - Chat completion with streaming + - `/v1/models` - List available models + - `/v1/completions` - Text completion (legacy) +- **Key Features**: + - Streaming with Server-Sent Events + - Message format compatibility + - Response structure compatibility + +## FastAPI Best Practices + +This project follows best practices from [github.com/zhanymkanov/fastapi-best-practices](https://github.com/zhanymkanov/fastapi-best-practices) + +### Project Structure + +**Domain-Based Organization**: Code is organized by domain/feature rather than by file type: + +``` +src/ +β”œβ”€β”€ chat/ # Chat completions domain +β”‚ β”œβ”€β”€ router.py # FastAPI routes +β”‚ β”œβ”€β”€ schemas.py # Pydantic request/response models +β”‚ β”œβ”€β”€ service.py # Business logic +β”‚ β”œβ”€β”€ dependencies.py # Domain-specific dependencies +β”‚ β”œβ”€β”€ constants.py # Domain constants +β”‚ └── __init__.py +β”œβ”€β”€ models/ # Models listing domain +β”‚ β”œβ”€β”€ router.py +β”‚ β”œβ”€β”€ schemas.py +β”‚ β”œβ”€β”€ service.py +β”‚ └── __init__.py +β”œβ”€β”€ core/ # Shared utilities +β”‚ β”œβ”€β”€ config.py # Global configuration +β”‚ β”œβ”€β”€ models.py # Custom base Pydantic models +β”‚ β”œβ”€β”€ exceptions.py # Global exceptions +β”‚ β”œβ”€β”€ dependencies.py # Shared dependencies +β”‚ └── router.py # Core routes (health, root) +β”œβ”€β”€ ollama/ # Ollama client layer +β”‚ β”œβ”€β”€ client.py # Async Ollama HTTP client +β”‚ β”œβ”€β”€ schemas.py # Ollama API models +β”‚ └── __init__.py +└── main.py # Application factory & configuration +``` + +**Key Principles**: +- Each domain has its own router, schemas, models, service, etc. +- Cross-domain imports use explicit naming: `from src.auth import constants as auth_constants` +- Main.py focuses on configuration, middleware, and exception handlers +- Business logic stays in service modules +- Routes delegate to services for all business logic + +### Async/Await Best Practices + +**Critical Understanding**: FastAPI handles sync and async routes differently: + +- **Async routes** (`async def`): Called directly in event loop + - Use ONLY for non-blocking operations + - Perfect for `await httpx.get()`, database queries, file I/O + - **NEVER** use blocking calls like `time.sleep()` - this blocks entire server + +- **Sync routes** (`def`): Run in thread pool + - Use for CPU-intensive work or blocking SDKs + - Blocking I/O won't freeze the event loop + - Example: `time.sleep(10)` is safe here + +**Example**: +```python +@router.get("/terrible") +async def terrible(): + time.sleep(10) # ❌ BLOCKS ENTIRE SERVER + +@router.get("/good") +def good(): + time.sleep(10) # βœ… Runs in thread pool + +@router.get("/perfect") +async def perfect(): + await asyncio.sleep(10) # βœ… Non-blocking async +``` + +**For CPU-intensive tasks**: Use separate worker processes (not threads) due to Python's GIL. + +### Pydantic Configuration + +**Custom Base Model**: All schemas inherit from `CustomBaseModel` for consistent behavior: + +```python +# src/core/models.py +class CustomBaseModel(BaseModel): + model_config = ConfigDict( + json_encoders={datetime: datetime_to_iso_str}, + populate_by_name=True, + use_enum_values=True, + validate_assignment=True, + ) + + def serializable_dict(self, **kwargs): + """Return dict with only JSON-serializable fields.""" + return jsonable_encoder(self.model_dump(**kwargs)) +``` + +**Benefits**: +- Consistent datetime serialization across all responses +- Alias support for field name flexibility +- Easy JSON encoding for logging/debugging + +**Decoupled Settings**: Split configuration by domain instead of one monolithic file: + +```python +# src/core/config.py - Global settings +class Config(BaseSettings): + DATABASE_URL: PostgresDsn + ENVIRONMENT: Environment + +# src/chat/config.py - Chat-specific settings +class ChatConfig(BaseSettings): + MAX_TOKENS: int + DEFAULT_TEMPERATURE: float +``` + +### Dependency Injection Patterns + +**Validation with Dependencies**: Use dependencies for complex validations: + +```python +async def valid_post_id(post_id: UUID4) -> dict: + """Validate post exists in database.""" + post = await service.get_by_id(post_id) + if not post: + raise PostNotFound() + return post + +@router.get("/posts/{post_id}") +async def get_post(post: dict = Depends(valid_post_id)): + return post # Already validated! +``` + +**Chaining Dependencies**: Build reusable validation layers: + +```python +async def valid_owned_post( + post: dict = Depends(valid_post_id), + token_data: dict = Depends(parse_jwt_data), +) -> dict: + if post["creator_id"] != token_data["user_id"]: + raise UserNotOwner() + return post +``` + +**Dependency Caching**: Dependencies are cached within request scope - FastAPI only executes each dependency once per request, even if used multiple times. + +### Application Factory Pattern + +Main.py uses factory pattern for testability and configuration: + +```python +def create_application() -> FastAPI: + """Create and configure FastAPI app.""" + app = FastAPI(title=config.APP_NAME) + + # Add middleware + app.add_middleware(CORSMiddleware, ...) + + # Register exception handlers + register_exception_handlers(app) + + # Include routers + app.include_router(chat_router, prefix="/v1") + + return app + +app = create_application() +``` + +## Development Guidelines + +### Code Structure +- Use async/await for ALL I/O operations (database, HTTP, file access) +- Use sync (def) for blocking SDKs or CPU-intensive work +- Implement proper error handling and logging +- Follow dependency injection for validation and shared resources +- Use Pydantic models for ALL request/response validation +- Keep business logic in service modules, not routers + +### Security Considerations +- Validate all inputs using Pydantic models +- Implement rate limiting for API endpoints +- Use environment variables for sensitive configuration +- Keep dependencies updated (check for CVEs regularly) + +### Testing +- Write integration tests for API endpoints +- Test streaming functionality thoroughly +- Mock Ollama responses for unit tests +- Validate OpenAI API compatibility + +### Configuration +- Use `.env` files for local development +- Document all environment variables in README +- Provide sensible defaults where possible +- Support container-based configuration + +## Common Patterns + +### Streaming Response Pattern +```python +from sse_starlette.sse import EventSourceResponse +from fastapi import FastAPI + +async def event_generator(): + # Stream events from Ollama/PydanticAI + yield {"data": "chunk1"} + yield {"data": "chunk2"} + +@app.get("/stream") +async def stream(): + return EventSourceResponse(event_generator()) +``` + +### PydanticAI Agent Pattern +```python +from pydantic_ai import Agent + +agent = Agent( + 'ollama:llama2', # Or other Ollama model + # Configuration here +) + +# Use the agent +result = await agent.run('Your prompt') +``` + +### OpenAI-Compatible Response Format +```python +{ + "id": "chatcmpl-123", + "object": "chat.completion.chunk", + "created": 1234567890, + "model": "model-name", + "choices": [{ + "index": 0, + "delta": {"content": "response"}, + "finish_reason": None + }] +} +``` + +## Update Policy + +This document should be updated when: +- Package versions are upgraded +- New major features are added +- Breaking API changes occur +- Security vulnerabilities are discovered + +Last updated: 2025-12-05 diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..6a5c789 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,45 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Added +- Initial project structure with domain-based organization +- OpenAI-compatible `/v1/chat/completions` endpoint with streaming support +- OpenAI-compatible `/v1/models` endpoint +- Health check and root endpoints +- Comprehensive test suite with 62% coverage +- Security-focused dependency management with CVE checking +- FastAPI best practices implementation +- SSE streaming with 20-second timeout protection +- Custom Pydantic base models for consistent serialization +- Application factory pattern for testability +- Async Ollama client (ready for integration) +- Complete documentation (README, AGENTS.md) + +### Security +- Minor version locking for all dependencies +- All packages CVE-checked (as of 2025-12-06) +- Environment variable protection via .gitignore +- No known vulnerabilities in dependency tree + +## [0.1.0] - 2025-12-06 + +### Added +- Project initialization +- Python 3.12.11 environment +- FastAPI 0.123.9 web framework +- PydanticAI 1.27.0 for LLM integration +- Mock chat completions (lorem ipsum responses) +- Mock model listing (mistral-nemo:latest) +- Testing infrastructure (pytest, coverage, ruff, mypy) +- Configuration management with pydantic-settings +- CORS middleware +- Exception handlers (OpenAI-compatible error format) + +[Unreleased]: https://github.com/yourusername/tatlock/compare/v0.1.0...HEAD +[0.1.0]: https://github.com/yourusername/tatlock/releases/tag/v0.1.0 diff --git a/README.md b/README.md new file mode 100644 index 0000000..ac51b34 --- /dev/null +++ b/README.md @@ -0,0 +1,304 @@ +# OpenAI-Compatible API with Ollama Backend + +A FastAPI-based service that provides an OpenAI-compatible API endpoint, powered by PydanticAI and Ollama for LLM inference. + +## Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Client │─────▢│ FastAPI Server │─────▢│ Ollama β”‚ +β”‚ │◀─────│ (Stream Coord.) │◀─────│ Container β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚Pydantic β”‚ + β”‚ AI β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Components + +- **FastAPI**: High-performance web framework providing the API layer +- **Stream Coordinator**: Manages streaming responses in OpenAI-compatible format +- **PydanticAI**: Agent framework handling LLM integration and structured outputs +- **Ollama**: External LLM backend (networked, managed separately) +- **SSE-Starlette**: Server-Sent Events for streaming responses + +## Features + +- OpenAI-compatible API endpoints +- Streaming responses with Server-Sent Events +- PydanticAI integration for robust LLM interactions +- Networked Ollama support +- Type-safe request/response handling with Pydantic +- Async/await throughout for optimal performance + +## Requirements + +- Python 3.12+ (Python 3.12.11 recommended for security) +- Network access to an existing Ollama instance (managed externally) + +## Installation + +### 1. Clone the repository + +```bash +git clone +cd tatlock +``` + +### 2. Create a virtual environment + +```bash +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +``` + +### 3. Install dependencies + +```bash +pip install -r requirements.txt +``` + +### 4. Configure environment variables + +Create a `.env` file in the project root: + +```env +# Ollama Configuration (point to your existing Ollama instance) +OLLAMA_HOST=http://your-ollama-host:11434 +OLLAMA_MODEL=llama2 + +# API Configuration +API_HOST=0.0.0.0 +API_PORT=8000 +API_RELOAD=true + +# Logging +LOG_LEVEL=info +``` + +**Note**: Update `OLLAMA_HOST` to point to your existing Ollama instance. Ensure the Ollama service is accessible from your network and has the required models installed. + +## Usage + +### Start the development server + +```bash +uvicorn main:app --reload +``` + +The API will be available at `http://localhost:8000` + +### API Endpoints + +#### Chat Completions (OpenAI-compatible) + +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "llama2", + "messages": [ + {"role": "user", "content": "Hello, how are you?"} + ], + "stream": true + }' +``` + +#### List Models + +```bash +curl http://localhost:8000/v1/models +``` + +### Interactive API Documentation + +- Swagger UI: `http://localhost:8000/docs` +- ReDoc: `http://localhost:8000/redoc` + +## Security + +### Version Locking Strategy + +This project uses minor version locking (`>=X.Y,