tatlock/AGENTS.md

LLM Agent Instructions

This document contains instructions and documentation references for AI assistants working with this codebase.

📖 Important: Before working on this project, read PHILOSOPHY.md to understand the system vision, architectural patterns, and design goals. All development should work towards realizing those patterns.

Project Overview

This project implements an OpenAI-compatible API with FastAPI, featuring a hybrid architecture that provides both the OpenAI Responses API and Chat Completions compatibility layer.

Architecture Pattern

The Orchestrator infrastructure layer with hybrid API architecture:

Client (Open WebUI)
      ↓
Chat Completions (/v1/chat/completions) → Wrapper
      ↓
Responses API (/v1/responses) → Primary
      ↓
Agent Interface (lorem-tester, Tatlock)
      ↓
Mock Agents (lorem-tester) / Future: PydanticAI Agents (Tatlock, Steward, etc.)

Architectural Layers:

1. The Orchestrator (Current Implementation)

   • FastAPI application providing the infrastructure
   • HTTP/SSE endpoints, streaming coordination
   • Conversation history and context management
   • OpenAI-compatible API surface

2. Future: The Household (Phases 1-4)

   • Steward: First-tier LLM for request analysis (PydanticAI agent)
   • Tatlock: Second-tier LLM with butler personality (PydanticAI agent)
   • Expert Agents: Domain specialists (Librarian, Developer, Handyman, etc.)

Key Architectural Decisions:

1. Single Source of Truth: All response generation happens in the Responses API

   • Structured output with reasoning, function_call, and message items
   • Real-time stop sequence and max tokens enforcement
   • Conversation history tracking
   • Context window management

2. Chat Completions Wrapper: Provides compatibility without duplicating logic

   • Calls Responses API internally
   • Automatically enables reasoning generation
   • Converts reasoning items to <think> tags for Open WebUI
   • Maintains OpenAI-compatible format

3. Agent Interface: Clean abstraction for multiple models

   • lorem-tester: Full-featured mock agent with realistic behavior
     • Reasoning summaries (adjustable effort levels)
     • Random tool/function calls
     • Error triggers for testing
     • Temperature variation
   • Tatlock: Advertised model name (currently mock, future: PydanticAI Butler agent)

4. Hybrid Conversation History:

   • Client MUST send full context in input array (OpenAI compatible)
   • Server optionally tracks via metadata.conversation_id
   • Auto-generates deterministic IDs from first message
   • Supports future vector memory integration (Qdrant)

Why This Architecture?

• Open WebUI Compatibility: Native Responses API support not yet in stable release
• Future-Proof: Easy migration when Open WebUI adds native support
• Testability: Full-featured mock agent (lorem-tester) for integration testing
• Clean Separation: Responses API as stable core, wrappers can change

Components

• FastAPI: Web framework for the API layer
• SSE-Starlette: Server-Sent Events for streaming responses
• Pydantic: Request/response validation with field validators
• Agent Interface: Abstract base class for model implementations
• Conversation History: Server-side tracking with configurable max turns
• Context Window: Token counting and management
• PydanticAI: Integrated with Tatlock agent (Ollama backend)
• Agent Tools: Permanent tools module (src/agents/tools.py)
  • Calculator: Safe mathematical expression evaluation
  • Date/Time toolkit: Current time, relative dates, time differences
  • Web Search: SearXNG integration for privacy-preserving search

Documentation References

Core Framework Documentation

FastAPI

• Official Documentation: https://fastapi.tiangolo.com/
• Version: 0.123.9 (Dec 2025)
• Key Topics:
  • Path operations and routing
  • Request/response models with Pydantic
  • Dependency injection
  • Background tasks
  • WebSocket and streaming support
• PyPI: https://pypi.org/project/fastapi/

Uvicorn

• Official Documentation: https://www.uvicorn.org/
• Version: 0.38.0 (Oct 2025)
• Key Topics:
  • ASGI server configuration
  • Deployment settings
  • Logging and monitoring
  • SSL/TLS configuration

AI/LLM Integration

PydanticAI

• Official Documentation: https://ai.pydantic.dev/
• Version: 1.27.0 (Dec 2025)
• Status: Dependency installed, ready for future integration
• Key Topics (for future implementation):
  • Agent creation and configuration
  • LLM provider integration (Ollama support)
  • Structured outputs with Pydantic
  • Streaming responses
  • Tool/function calling
  • RunContext and dynamic configuration
  • MCP server integration
• GitHub: https://github.com/pydantic/pydantic-ai
• PyPI: https://pypi.org/project/pydantic-ai/

Pydantic

• Official Documentation: https://docs.pydantic.dev/latest/
• Version: 2.11+ (Required for PydanticAI, currently using >=2.11,<2.13)
• Key Topics:
  • Data validation and serialization
  • Field types and validators
  • Model configuration
  • JSON schema generation

HTTP and Streaming

HTTPX

• Official Documentation: https://www.python-httpx.org/
• Version: 0.28.1
• Key Topics:
  • Async HTTP client for Ollama communication
  • Streaming responses
  • Timeout configuration
  • Connection pooling

SSE-Starlette

• GitHub: https://github.com/sysid/sse-starlette
• Version: 3.0.2 (Oct 2025)
• Key Topics:
  • Server-Sent Events implementation
  • Streaming event responses
  • Integration with FastAPI/Starlette

Ollama Integration

Ollama API

• Official Documentation: https://github.com/ollama/ollama/blob/main/docs/api.md
• Status: Async client implemented in src/ollama/client.py, ready for future integration
• Key Topics (for future implementation):
  • REST API endpoints
  • Streaming responses
  • Model management
  • Generate and chat endpoints
  • Model configuration
• Current Model Target: mistral-nemo:latest

OpenAI API Compatibility

OpenAI API Reference

• Official Documentation: https://platform.openai.com/docs/api-reference

• Key API Endpoints:

  • /v1/responses - Responses API (PRIMARY) with structured output
  • /v1/chat/completions - OpenAI Chat Completions compatibility wrapper
  • /v1/models - List available models

• Key Features for Development:

  • Responses API Format: Structured output with reasoning, function_call, and message items
  • Parameter Validation: Temperature, reasoning effort levels, max tokens, stop sequences
  • Conversation History: Hybrid client/server approach with auto-generated IDs
  • Context Management: Token counting and window trimming
  • Streaming: Real-time SSE streaming with stop sequence and max token enforcement
  • Error Handling: Custom exception types (RateLimitError, ContextLengthError)
  • Tool Calling: PydanticAI tool integration with permanent tools
  • Testing: Comprehensive test suite with mocks and real Ollama integration

FastAPI Best Practices

This project follows best practices from github.com/zhanymkanov/fastapi-best-practices

Project Structure

Domain-Based Organization: Code is organized by domain/feature rather than by file type:

src/
├── agents/                # Agent interface and implementations
│   ├── base.py            # Abstract AgentInterface
│   ├── lorem_tester.py    # Full-featured mock agent
│   ├── tatlock.py         # Placeholder for real agent
│   └── registry.py        # ModelRegistry for agent management
├── responses/             # Responses API domain (PRIMARY)
│   ├── router.py          # POST /v1/responses endpoint
│   ├── schemas.py         # Request/response models with validators
│   ├── service.py         # Response generation logic
│   ├── streaming.py       # SSE streaming coordinator
│   ├── history.py         # Conversation history management
│   └── context.py         # Context window and token management
├── chat/                  # Chat Completions domain (WRAPPER)
│   ├── router.py          # POST /v1/chat/completions endpoint
│   ├── schemas.py         # Chat request/response models
│   ├── service.py         # Wraps Responses API, converts to <think> tags
│   ├── constants.py       # Chat constants (roles, finish reasons)
│   └── __init__.py
├── models/                # Models listing domain
│   ├── router.py          # GET /v1/models endpoint
│   ├── schemas.py         # Model schemas
│   ├── service.py         # Accesses ModelRegistry
│   └── __init__.py
├── core/                  # Shared utilities
│   ├── config.py          # Global configuration (BaseSettings)
│   ├── models.py          # Custom base Pydantic models
│   ├── exceptions.py      # Custom exceptions (RateLimitError, etc.)
│   ├── dependencies.py    # Shared dependencies
│   └── router.py          # Core routes (health, root)
├── ollama/                # Ollama client layer (not yet integrated)
│   ├── client.py          # Async Ollama HTTP client
│   └── schemas.py         # Ollama API models
└── main.py                # Application factory & configuration

Key Architectural Principles:

• Single Source of Truth: Responses API handles all generation logic
• Wrapper Pattern: Chat Completions wraps Responses API without duplicating code
• Agent Abstraction: AgentInterface defines contract for all models
• Domain Separation: Each domain has its own router, schemas, service
• Service Layer: Business logic in services, not routers
• Type Safety: Pydantic models for ALL request/response validation
• Async First: All I/O operations use async/await

Async/Await Best Practices

Critical Understanding: FastAPI handles sync and async routes differently:

• Async routes (async def): Called directly in event loop

  • Use ONLY for non-blocking operations
  • Perfect for await httpx.get(), database queries, file I/O
  • NEVER use blocking calls like time.sleep() - this blocks entire server

• Sync routes (def): Run in thread pool

  • Use for CPU-intensive work or blocking SDKs
  • Blocking I/O won't freeze the event loop
  • Example: time.sleep(10) is safe here

Example:

@router.get("/terrible")
async def terrible():
    time.sleep(10)  # ❌ BLOCKS ENTIRE SERVER

@router.get("/good")
def good():
    time.sleep(10)  # ✅ Runs in thread pool

@router.get("/perfect")
async def perfect():
    await asyncio.sleep(10)  # ✅ Non-blocking async

For CPU-intensive tasks: Use separate worker processes (not threads) due to Python's GIL.

Pydantic Configuration

Custom Base Model: All schemas inherit from CustomBaseModel for consistent behavior:

# src/core/models.py
class CustomBaseModel(BaseModel):
    model_config = ConfigDict(
        json_encoders={datetime: datetime_to_iso_str},
        populate_by_name=True,
        use_enum_values=True,
        validate_assignment=True,
    )

    def serializable_dict(self, **kwargs):
        """Return dict with only JSON-serializable fields."""
        return jsonable_encoder(self.model_dump(**kwargs))

Benefits:

• Consistent datetime serialization across all responses
• Alias support for field name flexibility
• Easy JSON encoding for logging/debugging

Decoupled Settings: Split configuration by domain instead of one monolithic file:

# src/core/config.py - Global settings
class Config(BaseSettings):
    DATABASE_URL: PostgresDsn
    ENVIRONMENT: Environment

# src/chat/config.py - Chat-specific settings
class ChatConfig(BaseSettings):
    MAX_TOKENS: int
    DEFAULT_TEMPERATURE: float

Dependency Injection Patterns

Validation with Dependencies: Use dependencies for complex validations:

async def valid_post_id(post_id: UUID4) -> dict:
    """Validate post exists in database."""
    post = await service.get_by_id(post_id)
    if not post:
        raise PostNotFound()
    return post

@router.get("/posts/{post_id}")
async def get_post(post: dict = Depends(valid_post_id)):
    return post  # Already validated!

Chaining Dependencies: Build reusable validation layers:

async def valid_owned_post(
    post: dict = Depends(valid_post_id),
    token_data: dict = Depends(parse_jwt_data),
) -> dict:
    if post["creator_id"] != token_data["user_id"]:
        raise UserNotOwner()
    return post

Dependency Caching: Dependencies are cached within request scope - FastAPI only executes each dependency once per request, even if used multiple times.

Application Factory Pattern

Main.py uses factory pattern for testability and configuration:

def create_application() -> FastAPI:
    """Create and configure FastAPI app."""
    app = FastAPI(title=config.APP_NAME)

    # Add middleware
    app.add_middleware(CORSMiddleware, ...)

    # Register exception handlers
    register_exception_handlers(app)

    # Include routers
    app.include_router(chat_router, prefix="/v1")

    return app

app = create_application()

Development Guidelines

Git Workflow

IMPORTANT: Do NOT handle git commits or pushes automatically. Wait for explicit user instruction before:

• Running git add
• Running git commit
• Running git push
• Creating or pushing tags

The user will manage git operations themselves unless they specifically request assistance.

Server Logs and Debugging

Development Mode Logging: When the server is started using ./wakeup.sh, logs are written to logs/server.log. This file is:

• Cleared on each server startup (fresh logs every time)
• Written in real-time as the server runs
• Already gitignored (won't be committed)

Accessing Logs: You can read the log file at any time while the server is running:

# View current logs
cat logs/server.log

# Follow logs in real-time
tail -f logs/server.log

# Search logs
grep "ERROR" logs/server.log

This is useful for debugging issues, monitoring API calls, and understanding server behavior during development.

Code Structure Guidelines

• Use async/await for ALL I/O operations (database, HTTP, file access)
• Use sync (def) for blocking SDKs or CPU-intensive work
• Implement proper error handling and logging
• Follow dependency injection for validation and shared resources
• Use Pydantic models for ALL request/response validation
• Keep business logic in service modules, not routers
• Domain-based project structure (not file-type based)

Security Considerations

• Validate all inputs using Pydantic models
• Use environment variables for sensitive configuration
• Keep dependencies updated and CVE-checked
• Minor version locking for supply chain protection
• Consider rate limiting for production deployment
• Plan for authentication/API keys when needed

Testing Approach

• Write integration tests for API endpoints
• Test streaming functionality with appropriate timeouts
• Use pytest-asyncio for async test support
• Validate OpenAI API compatibility in tests
• Test both mock and real LLM integrations
• Cover main application (CORS, exception handlers, lifespan)
• Test wrapper layers (chat completions, etc.)
• Include tool functionality tests

Configuration Management

• Use .env files for local development
• Document all environment variables in README
• Provide sensible defaults where possible
• Use BaseSettings from pydantic-settings
• Support both local and container-based configuration

Common Patterns

Streaming Response Pattern

Example from src/chat/router.py:

from sse_starlette.sse import EventSourceResponse
from fastapi import FastAPI

async def event_generator():
    # Currently yields mock lorem ipsum chunks
    # Future: Stream from Ollama/PydanticAI
    yield {"data": chunk.model_dump_json()}
    yield {"data": "[DONE]"}

@app.post("/stream")
async def stream():
    return EventSourceResponse(event_generator())

PydanticAI Agent Pattern

When implementing agents with PydanticAI and Ollama:

from pydantic_ai import Agent

agent = Agent(
    'ollama:mistral-nemo',  # Target model
    # Configuration here
)

# Use the agent
result = await agent.run('Your prompt')

OpenAI-Compatible Response Format

Example schema from src/chat/schemas.py:

{
    "id": "chatcmpl-123",
    "object": "chat.completion.chunk",
    "created": 1234567890,
    "model": "mistral-nemo:latest",
    "choices": [{
        "index": 0,
        "delta": {"content": "response"},
        "finish_reason": None
    }]
}

PydanticAI Tool Registration Pattern

Tools are registered with PydanticAI agents using decorators. See src/agents/tatlock.py for examples:

from pydantic_ai import Agent, RunContext

# After creating the agent
@agent.tool
def tool_name(ctx: RunContext[None], param: str) -> str:
    """
    Tool description that the LLM sees.

    Args:
        param: Parameter description

    Returns:
        Result description
    """
    return result

Tool Implementation Guidelines:

• Keep tools in src/agents/tools.py for reusability
• Use clear, descriptive docstrings (LLM reads these)
• Include parameter descriptions in docstrings
• Handle errors gracefully and return error messages as strings
• For async operations, declare the tool function as async def
• Test tools independently before integration

Example Tool Module (src/agents/tools.py):

def calculate(expression: str) -> str:
    """Safe calculator implementation."""
    try:
        # Implementation
        return str(result)
    except Exception as e:
        return f"Error: {str(e)}"

async def search_web(query: str) -> str:
    """Web search via SearXNG."""
    async with httpx.AsyncClient() as client:
        # Implementation
        return formatted_results

Update Policy

This document should be updated when:

• New development patterns are established
• Package versions are upgraded
• Major architectural changes occur
• New best practices are identified

Last updated: 2025-12-06 (Tools integration)