# LLM Agent Instructions

This document contains instructions and documentation references for AI assistants working with this codebase.

> **📖 Important**: Before working on this project, read [PHILOSOPHY.md](PHILOSOPHY.md) to understand the system vision, architectural patterns, and design goals. All development should work towards realizing those patterns.

## Project Overview

This project implements an OpenAI-compatible API with FastAPI, featuring a hybrid architecture that provides both the OpenAI Responses API and Chat Completions compatibility layer.

### Architecture Pattern

The **Orchestrator** infrastructure layer with hybrid API architecture:

```
Client (Open WebUI)
      ↓
Chat Completions (/v1/chat/completions) → Wrapper
      ↓
Responses API (/v1/responses) → Primary
      ↓
Agent Interface (lorem-tester, Tatlock)
      ↓
Mock Agents (lorem-tester) / Future: PydanticAI Agents (Tatlock, Steward, etc.)
```

**Architectural Layers:**

1. **The Orchestrator** (Current Implementation)
   - FastAPI application providing the infrastructure
   - HTTP/SSE endpoints, streaming coordination
   - Conversation history and context management
   - OpenAI-compatible API surface

2. **Future: The Household** (Phases 1-4)
   - **Steward**: First-tier LLM for request analysis (PydanticAI agent)
   - **Tatlock**: Second-tier LLM with butler personality (PydanticAI agent)
   - **Expert Agents**: Domain specialists (Librarian, Developer, Handyman, etc.)

**Key Architectural Decisions:**

1. **Single Source of Truth**: All response generation happens in the Responses API
   - Structured output with reasoning, function_call, and message items
   - Real-time stop sequence and max tokens enforcement
   - Conversation history tracking
   - Context window management

2. **Chat Completions Wrapper**: Provides compatibility without duplicating logic
   - Calls Responses API internally
   - Automatically enables reasoning generation
   - Converts reasoning items to `<think>` tags for Open WebUI
   - Maintains OpenAI-compatible format

3. **Agent Interface**: Clean abstraction for multiple models
   - **lorem-tester**: Full-featured mock agent with realistic behavior
     - Reasoning summaries (adjustable effort levels)
     - Random tool/function calls
     - Error triggers for testing
     - Temperature variation
   - **Tatlock**: Advertised model name (currently mock, future: PydanticAI Butler agent)

4. **Hybrid Conversation History**:
   - Client MUST send full context in `input` array (OpenAI compatible)
   - Server optionally tracks via `metadata.conversation_id`
   - Auto-generates deterministic IDs from first message
   - Supports future vector memory integration (Qdrant)

**Why This Architecture?**

- **Open WebUI Compatibility**: Native Responses API support not yet in stable release
- **Future-Proof**: Easy migration when Open WebUI adds native support
- **Testability**: Full-featured mock agent (lorem-tester) for integration testing
- **Clean Separation**: Responses API as stable core, wrappers can change

### Components

- **FastAPI**: Web framework for the API layer
- **SSE-Starlette**: Server-Sent Events for streaming responses
- **Pydantic**: Request/response validation with field validators
- **Agent Interface**: Abstract base class for model implementations
- **Conversation History**: Server-side tracking with configurable max turns
- **Context Window**: Token counting and management
- **PydanticAI**: Integrated with Tatlock agent (Ollama backend)
- **Agent Tools**: Permanent tools module (`src/agents/tools.py`)
  - Calculator: Safe mathematical expression evaluation
  - Date/Time toolkit: Current time, relative dates, time differences
  - Web Search: SearXNG integration for privacy-preserving search

## Documentation References

### Core Framework Documentation

#### FastAPI
- **Official Documentation**: https://fastapi.tiangolo.com/
- **Version**: 0.123.9 (Dec 2025)
- **Key Topics**:
  - Path operations and routing
  - Request/response models with Pydantic
  - Dependency injection
  - Background tasks
  - WebSocket and streaming support
- **PyPI**: https://pypi.org/project/fastapi/

#### Uvicorn
- **Official Documentation**: https://www.uvicorn.org/
- **Version**: 0.38.0 (Oct 2025)
- **Key Topics**:
  - ASGI server configuration
  - Deployment settings
  - Logging and monitoring
  - SSL/TLS configuration

### AI/LLM Integration

#### PydanticAI
- **Official Documentation**: https://ai.pydantic.dev/
- **Version**: 1.27.0 (Dec 2025)
- **Status**: Dependency installed, ready for future integration
- **Key Topics** (for future implementation):
  - Agent creation and configuration
  - LLM provider integration (Ollama support)
  - Structured outputs with Pydantic
  - Streaming responses
  - Tool/function calling
  - RunContext and dynamic configuration
  - MCP server integration
- **GitHub**: https://github.com/pydantic/pydantic-ai
- **PyPI**: https://pypi.org/project/pydantic-ai/

#### Pydantic
- **Official Documentation**: https://docs.pydantic.dev/latest/
- **Version**: 2.11+ (Required for PydanticAI, currently using >=2.11,<2.13)
- **Key Topics**:
  - Data validation and serialization
  - Field types and validators
  - Model configuration
  - JSON schema generation

### HTTP and Streaming

#### HTTPX
- **Official Documentation**: https://www.python-httpx.org/
- **Version**: 0.28.1
- **Key Topics**:
  - Async HTTP client for Ollama communication
  - Streaming responses
  - Timeout configuration
  - Connection pooling

#### SSE-Starlette
- **GitHub**: https://github.com/sysid/sse-starlette
- **Version**: 3.0.2 (Oct 2025)
- **Key Topics**:
  - Server-Sent Events implementation
  - Streaming event responses
  - Integration with FastAPI/Starlette

### Ollama Integration

#### Ollama API
- **Official Documentation**: https://github.com/ollama/ollama/blob/main/docs/api.md
- **Status**: Async client implemented in `src/ollama/client.py`, ready for future integration
- **Key Topics** (for future implementation):
  - REST API endpoints
  - Streaming responses
  - Model management
  - Generate and chat endpoints
  - Model configuration
- **Current Model Target**: mistral-nemo:latest

### OpenAI API Compatibility

#### OpenAI API Reference
- **Official Documentation**: https://platform.openai.com/docs/api-reference
- **Key API Endpoints**:
  - `/v1/responses` - Responses API (PRIMARY) with structured output
  - `/v1/chat/completions` - OpenAI Chat Completions compatibility wrapper
  - `/v1/models` - List available models

- **Key Features for Development**:
  - **Responses API Format**: Structured output with reasoning, function_call, and message items
  - **Parameter Validation**: Temperature, reasoning effort levels, max tokens, stop sequences
  - **Conversation History**: Hybrid client/server approach with auto-generated IDs
  - **Context Management**: Token counting and window trimming
  - **Streaming**: Real-time SSE streaming with stop sequence and max token enforcement
  - **Error Handling**: Custom exception types (RateLimitError, ContextLengthError)
  - **Tool Calling**: PydanticAI tool integration with permanent tools
  - **Testing**: Comprehensive test suite with mocks and real Ollama integration

## FastAPI Best Practices

This project follows best practices from [github.com/zhanymkanov/fastapi-best-practices](https://github.com/zhanymkanov/fastapi-best-practices)

### Project Structure

**Domain-Based Organization**: Code is organized by domain/feature rather than by file type:

```
src/
├── agents/                # Agent interface and implementations
│   ├── base.py            # Abstract AgentInterface
│   ├── lorem_tester.py    # Full-featured mock agent
│   ├── tatlock.py         # Placeholder for real agent
│   └── registry.py        # ModelRegistry for agent management
├── responses/             # Responses API domain (PRIMARY)
│   ├── router.py          # POST /v1/responses endpoint
│   ├── schemas.py         # Request/response models with validators
│   ├── service.py         # Response generation logic
│   ├── streaming.py       # SSE streaming coordinator
│   ├── history.py         # Conversation history management
│   └── context.py         # Context window and token management
├── chat/                  # Chat Completions domain (WRAPPER)
│   ├── router.py          # POST /v1/chat/completions endpoint
│   ├── schemas.py         # Chat request/response models
│   ├── service.py         # Wraps Responses API, converts to <think> tags
│   ├── constants.py       # Chat constants (roles, finish reasons)
│   └── __init__.py
├── models/                # Models listing domain
│   ├── router.py          # GET /v1/models endpoint
│   ├── schemas.py         # Model schemas
│   ├── service.py         # Accesses ModelRegistry
│   └── __init__.py
├── core/                  # Shared utilities
│   ├── config.py          # Global configuration (BaseSettings)
│   ├── models.py          # Custom base Pydantic models
│   ├── exceptions.py      # Custom exceptions (RateLimitError, etc.)
│   ├── dependencies.py    # Shared dependencies
│   └── router.py          # Core routes (health, root)
├── ollama/                # Ollama client layer (not yet integrated)
│   ├── client.py          # Async Ollama HTTP client
│   └── schemas.py         # Ollama API models
└── main.py                # Application factory & configuration
```

**Key Architectural Principles**:
- **Single Source of Truth**: Responses API handles all generation logic
- **Wrapper Pattern**: Chat Completions wraps Responses API without duplicating code
- **Agent Abstraction**: AgentInterface defines contract for all models
- **Domain Separation**: Each domain has its own router, schemas, service
- **Service Layer**: Business logic in services, not routers
- **Type Safety**: Pydantic models for ALL request/response validation
- **Async First**: All I/O operations use async/await

### Async/Await Best Practices

**Critical Understanding**: FastAPI handles sync and async routes differently:

- **Async routes** (`async def`): Called directly in event loop
  - Use ONLY for non-blocking operations
  - Perfect for `await httpx.get()`, database queries, file I/O
  - **NEVER** use blocking calls like `time.sleep()` - this blocks entire server

- **Sync routes** (`def`): Run in thread pool
  - Use for CPU-intensive work or blocking SDKs
  - Blocking I/O won't freeze the event loop
  - Example: `time.sleep(10)` is safe here

**Example**:
```python
@router.get("/terrible")
async def terrible():
    time.sleep(10)  # ❌ BLOCKS ENTIRE SERVER

@router.get("/good")
def good():
    time.sleep(10)  # ✅ Runs in thread pool

@router.get("/perfect")
async def perfect():
    await asyncio.sleep(10)  # ✅ Non-blocking async
```

**For CPU-intensive tasks**: Use separate worker processes (not threads) due to Python's GIL.

### Pydantic Configuration

**Custom Base Model**: All schemas inherit from `CustomBaseModel` for consistent behavior:

```python
# src/core/models.py
class CustomBaseModel(BaseModel):
    model_config = ConfigDict(
        json_encoders={datetime: datetime_to_iso_str},
        populate_by_name=True,
        use_enum_values=True,
        validate_assignment=True,
    )

    def serializable_dict(self, **kwargs):
        """Return dict with only JSON-serializable fields."""
        return jsonable_encoder(self.model_dump(**kwargs))
```

**Benefits**:
- Consistent datetime serialization across all responses
- Alias support for field name flexibility
- Easy JSON encoding for logging/debugging

**Decoupled Settings**: Split configuration by domain instead of one monolithic file:

```python
# src/core/config.py - Global settings
class Config(BaseSettings):
    DATABASE_URL: PostgresDsn
    ENVIRONMENT: Environment

# src/chat/config.py - Chat-specific settings
class ChatConfig(BaseSettings):
    MAX_TOKENS: int
    DEFAULT_TEMPERATURE: float
```

### Dependency Injection Patterns

**Validation with Dependencies**: Use dependencies for complex validations:

```python
async def valid_post_id(post_id: UUID4) -> dict:
    """Validate post exists in database."""
    post = await service.get_by_id(post_id)
    if not post:
        raise PostNotFound()
    return post

@router.get("/posts/{post_id}")
async def get_post(post: dict = Depends(valid_post_id)):
    return post  # Already validated!
```

**Chaining Dependencies**: Build reusable validation layers:

```python
async def valid_owned_post(
    post: dict = Depends(valid_post_id),
    token_data: dict = Depends(parse_jwt_data),
) -> dict:
    if post["creator_id"] != token_data["user_id"]:
        raise UserNotOwner()
    return post
```

**Dependency Caching**: Dependencies are cached within request scope - FastAPI only executes each dependency once per request, even if used multiple times.

### Application Factory Pattern

Main.py uses factory pattern for testability and configuration:

```python
def create_application() -> FastAPI:
    """Create and configure FastAPI app."""
    app = FastAPI(title=config.APP_NAME)

    # Add middleware
    app.add_middleware(CORSMiddleware, ...)

    # Register exception handlers
    register_exception_handlers(app)

    # Include routers
    app.include_router(chat_router, prefix="/v1")

    return app

app = create_application()
```

## Development Guidelines

### Git Workflow

**IMPORTANT**: Do NOT handle git commits or pushes automatically. Wait for explicit user instruction before:
- Running `git add`
- Running `git commit`
- Running `git push`
- Creating or pushing tags

The user will manage git operations themselves unless they specifically request assistance.

### Server Logs and Debugging

**Development Mode Logging**: When the server is started using `./wakeup.sh`, logs are written to `logs/server.log`. This file is:
- Cleared on each server startup (fresh logs every time)
- Written in real-time as the server runs
- Already gitignored (won't be committed)

**Accessing Logs**: You can read the log file at any time while the server is running:
```bash
# View current logs
cat logs/server.log

# Follow logs in real-time
tail -f logs/server.log

# Search logs
grep "ERROR" logs/server.log
```

This is useful for debugging issues, monitoring API calls, and understanding server behavior during development.

### Code Structure Guidelines
- Use async/await for ALL I/O operations (database, HTTP, file access)
- Use sync (def) for blocking SDKs or CPU-intensive work
- Implement proper error handling and logging
- Follow dependency injection for validation and shared resources
- Use Pydantic models for ALL request/response validation
- Keep business logic in service modules, not routers
- Domain-based project structure (not file-type based)

### Security Considerations
- Validate all inputs using Pydantic models
- Use environment variables for sensitive configuration
- Keep dependencies updated and CVE-checked
- Minor version locking for supply chain protection
- Consider rate limiting for production deployment
- Plan for authentication/API keys when needed

### Testing Approach
- Write integration tests for API endpoints
- Test streaming functionality with appropriate timeouts
- Use pytest-asyncio for async test support
- Validate OpenAI API compatibility in tests
- Test both mock and real LLM integrations
- Cover main application (CORS, exception handlers, lifespan)
- Test wrapper layers (chat completions, etc.)
- Include tool functionality tests

### Configuration Management
- Use `.env` files for local development
- Document all environment variables in README
- Provide sensible defaults where possible
- Use BaseSettings from pydantic-settings
- Support both local and container-based configuration

## Common Patterns

### Streaming Response Pattern

Example from `src/chat/router.py`:

```python
from sse_starlette.sse import EventSourceResponse
from fastapi import FastAPI

async def event_generator():
    # Currently yields mock lorem ipsum chunks
    # Future: Stream from Ollama/PydanticAI
    yield {"data": chunk.model_dump_json()}
    yield {"data": "[DONE]"}

@app.post("/stream")
async def stream():
    return EventSourceResponse(event_generator())
```

### PydanticAI Agent Pattern

When implementing agents with PydanticAI and Ollama:

```python
from pydantic_ai import Agent

agent = Agent(
    'ollama:mistral-nemo',  # Target model
    # Configuration here
)

# Use the agent
result = await agent.run('Your prompt')
```

### OpenAI-Compatible Response Format

Example schema from `src/chat/schemas.py`:

```python
{
    "id": "chatcmpl-123",
    "object": "chat.completion.chunk",
    "created": 1234567890,
    "model": "mistral-nemo:latest",
    "choices": [{
        "index": 0,
        "delta": {"content": "response"},
        "finish_reason": None
    }]
}
```

### PydanticAI Tool Registration Pattern

Tools are registered with PydanticAI agents using decorators. See `src/agents/tatlock.py` for examples:

```python
from pydantic_ai import Agent, RunContext

# After creating the agent
@agent.tool
def tool_name(ctx: RunContext[None], param: str) -> str:
    """
    Tool description that the LLM sees.

    Args:
        param: Parameter description

    Returns:
        Result description
    """
    return result
```

**Tool Implementation Guidelines**:
- Keep tools in `src/agents/tools.py` for reusability
- Use clear, descriptive docstrings (LLM reads these)
- Include parameter descriptions in docstrings
- Handle errors gracefully and return error messages as strings
- For async operations, declare the tool function as `async def`
- Test tools independently before integration

**Example Tool Module** (`src/agents/tools.py`):
```python
def calculate(expression: str) -> str:
    """Safe calculator implementation."""
    try:
        # Implementation
        return str(result)
    except Exception as e:
        return f"Error: {str(e)}"

async def search_web(query: str) -> str:
    """Web search via SearXNG."""
    async with httpx.AsyncClient() as client:
        # Implementation
        return formatted_results
```

## Update Policy

This document should be updated when:
- New development patterns are established
- Package versions are upgraded
- Major architectural changes occur
- New best practices are identified

Last updated: 2025-12-06 (Tools integration)