refactor: consolidate project structure and clean up documentation
- Move docs to docs/ (philosophy, roadmap, orchestration scenarios, claude integration, testing improvements) - Strip completed phases from roadmap and claude integration docs - Move dependencies from requirements*.txt into pyproject.toml - Move pytest config from pytest.ini into pyproject.toml - Add Makefile replacing wakeup.sh (setup, run, test, lint, etc.) - Add CI test gate in Gitea Actions workflow - Consolidate caches into .cache/ (pytest, mypy, ruff) - Consolidate build output into build/ (coverage, logs) - Update Dockerfile for pyproject.toml install - Update cross-references in README, AGENTS.md, CLAUDE.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.6
@@ -0,0 +1,100 @@
|
||||
# Claude Integration Plan
|
||||
|
||||
## Overview
|
||||
|
||||
Tatlock uses a bidirectional Claude architecture:
|
||||
- **Scenario A**: Tatlock powered by Claude backend (with Ollama fallback) — **COMPLETE**
|
||||
- **Scenario B**: Tatlock exposed as MCP server for external Claude instances — **OPEN**
|
||||
- **Scenario C**: Offline operation via Ollama — **COMPLETE**
|
||||
|
||||
---
|
||||
|
||||
## MCP Server (Expose Tools to Claude) — NOT STARTED
|
||||
|
||||
Create an MCP server that exposes Tatlock's household tools to external Claude instances.
|
||||
|
||||
### New Files
|
||||
|
||||
```
|
||||
src/mcp/
|
||||
├── __init__.py
|
||||
├── server.py # MCP server using mcp Python SDK
|
||||
├── tool_adapters.py # Convert PydanticAI tools → MCP schemas
|
||||
├── auth.py # API key authentication
|
||||
└── transport.py # Streamable HTTP transport
|
||||
```
|
||||
|
||||
### Docker Stack Addition
|
||||
|
||||
```yaml
|
||||
tatlock-mcp:
|
||||
image: git.schweitz.internal/jpmschweitzer/tatlock:latest
|
||||
command: ["python", "-m", "src.mcp.server"]
|
||||
ports:
|
||||
- "8002:8002"
|
||||
environment:
|
||||
- MCP_AUTH_TOKEN=${MCP_AUTH_TOKEN}
|
||||
networks:
|
||||
- docker-dataplane
|
||||
```
|
||||
|
||||
### Claude Desktop Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"tatlock": {
|
||||
"command": "npx",
|
||||
"args": ["mcp-remote", "https://mcp.schweitz.net/sse", "--header", "Authorization: Bearer ${MCP_AUTH_TOKEN}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Checklist
|
||||
|
||||
- [ ] Create `src/mcp/` module
|
||||
- [ ] Tool adapters (PydanticAI → MCP schema)
|
||||
- [ ] Authentication middleware
|
||||
- [ ] Streamable HTTP transport
|
||||
- [ ] Docker stack configuration
|
||||
|
||||
---
|
||||
|
||||
## Future Phases
|
||||
|
||||
- **LiteLLM Gateway** — Unified endpoint for all models, config-driven routing
|
||||
- **Multi-Provider** — Add OpenAI, Vertex AI, etc.
|
||||
- **Smart Routing** — Context-aware model selection, cost ceiling enforcement
|
||||
|
||||
---
|
||||
|
||||
## Offline Behavior
|
||||
|
||||
| Scenario | Behavior |
|
||||
|----------|----------|
|
||||
| No API key | Use Ollama exclusively |
|
||||
| API unreachable | Use Ollama, log warning |
|
||||
| API rate limited | Fallback to Ollama |
|
||||
|
||||
| Aspect | Claude | Ollama |
|
||||
|--------|--------|--------|
|
||||
| Context | 200k tokens | ~8k tokens |
|
||||
| Latency | 1-3s (network) | 0.5-1s (local) |
|
||||
| Personality | Preserved | Preserved |
|
||||
| Tools | All work | All work |
|
||||
| Cost | API charges | Free |
|
||||
|
||||
---
|
||||
|
||||
## Related Repo Handovers
|
||||
|
||||
Handover documents created in each repo: `PROJECT_CLAUDIFICATION_HANDOVER.md`
|
||||
|
||||
### Open Items
|
||||
|
||||
- **library-desk**: Review HybridRAG response size limits, smart_create endpoint, response formats
|
||||
- **core-api**: Review list_devices response format, error messages, rate limiting
|
||||
- **portainer-core**: Update stack with new env vars, configure secrets, update CONTAINERS.md
|
||||
- **webber**: Review content truncation limits, extraction quality
|
||||
- **tatlock-ui**: Test streaming with Claude backend, conversation history, tool call display
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,278 @@
|
||||
# Tatlock - System Philosophy and Architecture
|
||||
|
||||
## Document Purpose
|
||||
|
||||
This document establishes the foundational philosophy and architectural patterns for the Tatlock system. It represents the **target design** that all development should work towards.
|
||||
|
||||
**When to modify this document**:
|
||||
- When there is a deliberate decision to deviate from these established patterns
|
||||
- When fundamental assumptions about the system's purpose change
|
||||
- When new architectural insights require rethinking core principles
|
||||
|
||||
**When NOT to modify this document**:
|
||||
- During implementation of these patterns (use README.md, AGENTS.md, or code comments for technical details)
|
||||
- For adding new household members or capabilities within the existing pattern
|
||||
- For tactical decisions about specific technologies or tools
|
||||
|
||||
This document should remain stable, serving as the north star for development decisions.
|
||||
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
### Vision
|
||||
|
||||
Tatlock is a comprehensive homelab butler and personal assistant system designed to augment personal and household productivity through intelligent automation, knowledge management, and contextual assistance. Named after a traditional British butler, Tatlock embodies the wit, competence, and organizational skill of a well-run household staff, coordinating a team of specialized expert agents to serve the needs of its users.
|
||||
|
||||
Unlike cloud-dependent AI assistants, Tatlock is built to operate primarily offline, maintaining privacy and control while providing sophisticated assistance across multiple domains of daily life.
|
||||
|
||||
### Purpose
|
||||
|
||||
The system serves as a unified intelligent interface for:
|
||||
|
||||
- **Knowledge Work**: Research assistance, information synthesis, general knowledge queries
|
||||
- **Technical Work**: Software development support, systems administration tasks
|
||||
- **Home Management**: Home automation control and monitoring
|
||||
- **Personal Organization**: Calendaring, scheduling, task management, list keeping
|
||||
- **Information Management**: Personal documentation, note-taking, knowledge base maintenance
|
||||
|
||||
### Core Philosophy
|
||||
|
||||
Tatlock is built on three fundamental principles:
|
||||
|
||||
1. **Privacy-First Architecture**: All processing occurs locally within your homelab environment. Your data, conversations, and personal information never leave your infrastructure unless you explicitly direct it to do so.
|
||||
|
||||
2. **Offline-Capable Operation**: While the system can leverage internet resources when available, core functionality remains operational without external connectivity. This ensures reliability and independence from third-party services.
|
||||
|
||||
3. **Multi-Tenant by Design**: Though primarily intended for personal use (yourself, household members, and close friends), the system architecture supports multiple users with complete data isolation, personalized experiences, and individual preferences.
|
||||
|
||||
### Scope
|
||||
|
||||
**Current Focus**: The initial implementation establishes the foundational architecture with OpenAI-compatible API interfaces, structured response formats, and reasoning transparency. This phase prioritizes:
|
||||
- Core API infrastructure
|
||||
- Response streaming and formatting
|
||||
- Basic conversation management
|
||||
- Testing and validation framework
|
||||
|
||||
**Future Expansion**: The system will evolve into a comprehensive personal assistant platform by integrating:
|
||||
- Specialized containerized services (machine learning, search, storage, memory)
|
||||
- Task and project management capabilities
|
||||
- Calendar and scheduling systems
|
||||
- Home automation integration
|
||||
- Personal knowledge management
|
||||
- Advanced multi-agent collaboration
|
||||
|
||||
### Deployment Model
|
||||
|
||||
Tatlock is designed for **single-instance, multi-user deployment** within a homelab environment:
|
||||
|
||||
- **Users**: Personal use for household members and trusted friends
|
||||
- **Infrastructure**: Self-hosted on your own hardware
|
||||
- **Architecture**: Containerized microservices on a single host
|
||||
- **Data Sovereignty**: Complete control over all data and processing
|
||||
|
||||
This deployment model balances simplicity of operation with the security and personalization needs of a small, trusted user base.
|
||||
|
||||
### System Context
|
||||
|
||||
Tatlock operates as the central orchestration layer within a broader ecosystem of containerized services:
|
||||
|
||||
#### Core Service Stack
|
||||
- **Language Models**: Ollama for local ML inference
|
||||
- **Search**: SearxNG for privacy-respecting web search
|
||||
- **Memory Systems**:
|
||||
- Redis for short-term memory and caching
|
||||
- Qdrant for long-term memory and vector storage
|
||||
- **Data Storage**: PostgreSQL for structured data and multi-tenant isolation
|
||||
- **Future Services**: Calendaring, scheduling, task management, documentation systems
|
||||
|
||||
#### Integration Approach
|
||||
Rather than building monolithic functionality, Tatlock acts as an intelligent coordinator, leveraging specialized services for specific capabilities while maintaining consistent interfaces and user experience.
|
||||
|
||||
### Design Goals
|
||||
|
||||
1. **Unified Experience**: Single point of interaction for diverse personal assistance needs
|
||||
2. **Contextual Intelligence**: Understanding across conversations, tasks, and time
|
||||
3. **Transparent Operation**: Visible reasoning and decision-making processes
|
||||
4. **Extensible Architecture**: Easy integration of new capabilities and services
|
||||
5. **Reliable Performance**: Consistent operation regardless of internet availability
|
||||
6. **User Privacy**: Zero data leakage to external parties
|
||||
7. **Multi-User Support**: Isolated experiences for different household members
|
||||
|
||||
### Success Criteria
|
||||
|
||||
Tatlock succeeds when it becomes the natural first point of interaction for:
|
||||
- Answering questions and conducting research
|
||||
- Managing daily tasks and schedules
|
||||
- Controlling home automation
|
||||
- Supporting development and technical work
|
||||
- Organizing personal information and knowledge
|
||||
|
||||
The system should feel less like "using a tool" and more like "asking a capable assistant" who understands your context, preferences, and needs.
|
||||
|
||||
## The Household Architecture
|
||||
|
||||
### System Layers
|
||||
|
||||
The Tatlock system consists of two distinct architectural layers:
|
||||
|
||||
#### The Orchestrator (Infrastructure Layer)
|
||||
|
||||
The **Orchestrator** is the FastAPI application that provides the technical infrastructure:
|
||||
- HTTP/SSE endpoints (`/v1/responses`, `/v1/chat/completions`)
|
||||
- Streaming coordination and conversation management
|
||||
- Token counting and context window management
|
||||
- Integration with Open WebUI and other clients
|
||||
- Request/response lifecycle management
|
||||
|
||||
This is the "plumbing" layer that exists now and handles all the technical concerns of running an OpenAI-compatible API.
|
||||
|
||||
#### Tatlock - The Butler (Agent Layer)
|
||||
|
||||
**Tatlock** is the PydanticAI agent that provides the intelligence and personality:
|
||||
- The witty British butler persona
|
||||
- Coordination with the Steward and household staff
|
||||
- Multi-agent orchestration and synthesis
|
||||
- Context-aware, personalized responses
|
||||
|
||||
The Orchestrator hosts Tatlock—users interact with "Tatlock" (the advertised model name), but technically they're talking to the Orchestrator infrastructure which routes requests through the Tatlock agent.
|
||||
|
||||
**Current State**: The Orchestrator exists and uses mock agents. Phase 1-3 of the implementation roadmap will integrate the real Tatlock agent using PydanticAI.
|
||||
|
||||
### The British Household Metaphor
|
||||
|
||||
Tatlock adopts the organizational structure of a traditional British estate household, where specialized staff members handle distinct domains of responsibility under the coordination of a capable butler. This metaphor is not merely aesthetic—it reflects a deliberate architectural pattern that enables focused expertise, clear separation of concerns, and efficient coordination.
|
||||
|
||||
### Household Roles
|
||||
|
||||
#### Tatlock - The Butler (Primary Interface)
|
||||
|
||||
**Character**: Witty, capable, and impeccably organized
|
||||
**Role**: Chief coordinator and primary point of contact with users
|
||||
|
||||
Tatlock serves as the face of the system, managing all user interactions with personality and competence. He understands the full context of requests, coordinates with appropriate household staff, synthesizes their contributions, and delivers coherent, thoughtful responses. His wit and personality make interactions engaging while maintaining professionalism.
|
||||
|
||||
**Responsibilities**:
|
||||
- Receiving and understanding user requests
|
||||
- Coordinating with household staff (expert agents)
|
||||
- Synthesizing multi-source information into coherent responses
|
||||
- Maintaining conversation context and user preferences
|
||||
- Presenting results with appropriate personality and tone
|
||||
|
||||
#### The Steward (Request Analysis)
|
||||
|
||||
**Role**: Initial request triage and resource planning
|
||||
|
||||
Before Tatlock engages with a request, the Steward performs crucial preparatory work. The Steward analyzes incoming requests to determine which tools, services, and household staff members will be needed, creating a curated recommendation that streamlines Tatlock's work.
|
||||
|
||||
**Responsibilities**:
|
||||
- Analyzing user requests for required capabilities
|
||||
- Identifying relevant tools and expert agents
|
||||
- Providing recommendations to focus Tatlock's attention
|
||||
- Reducing cognitive load on the Butler by pre-filtering options
|
||||
|
||||
#### Expert Household Staff (Domain Specialists)
|
||||
|
||||
**The Handyman** - System Maintenance and Technical Operations
|
||||
Handles system administration, server management, infrastructure monitoring, and technical troubleshooting.
|
||||
|
||||
**The Housekeeper** - Home Automation Management
|
||||
Controls and monitors home automation systems, environmental controls, security, and physical space management.
|
||||
|
||||
**The Secretary** - Scheduling and Organization
|
||||
Manages calendars, appointments, scheduling conflicts, reminders, and time-based coordination.
|
||||
|
||||
**The Developer** - Software Development Support
|
||||
Assists with code writing, debugging, architecture decisions, documentation, and development workflows.
|
||||
|
||||
**Additional Staff** (Future):
|
||||
- The Librarian - Knowledge management and research
|
||||
- The Accountant - Financial tracking and analysis
|
||||
- The Chef - Meal planning and nutrition
|
||||
- Others as needs emerge
|
||||
|
||||
### The Two-Tier Request Flow
|
||||
|
||||
The household operates through a carefully orchestrated two-tier process:
|
||||
|
||||
#### Tier 1: The Steward's Preparation
|
||||
|
||||
1. **User request arrives** at the Orchestrator (via HTTP API)
|
||||
2. **Orchestrator routes** the raw request to the Steward for analysis
|
||||
3. **Steward determines** which tools and household staff are relevant
|
||||
4. **Steward prepares recommendations**, written as a note to Tatlock
|
||||
5. **Recommendations are prepended** to the user's request
|
||||
|
||||
**Purpose**: This separation ensures that Tatlock isn't overwhelmed with the full universe of available tools and agents. The Steward narrows the scope to only relevant capabilities, making Tatlock's decision-making cleaner and more focused.
|
||||
|
||||
#### Tier 2: Tatlock's Orchestration
|
||||
|
||||
1. **Tatlock receives** the enriched request (original + Steward's notes)
|
||||
2. **Scope is limited** to recommended tools and staff only
|
||||
3. **Tatlock coordinates** with appropriate household members
|
||||
4. **Expert agents perform** their specialized tasks
|
||||
5. **All interactions are streamed** to the reasoning output in real-time
|
||||
6. **Tatlock synthesizes** results into a coherent response
|
||||
7. **User receives** a unified answer from Tatlock
|
||||
|
||||
**Purpose**: This tier focuses on execution and coordination. With a curated set of tools, Tatlock can efficiently orchestrate multiple expert agents, combine their outputs, and present a seamless response to the user.
|
||||
|
||||
**Real-Time Transparency**: Every interaction—whether Tatlock consulting the Handyman, waiting for a database query, or receiving results from the Secretary—is piped directly into the orchestrator's reasoning output. Users see the household at work in real-time, understanding what's happening even when operations take time. This transforms potentially frustrating wait times into engaging insight into the system's thought process.
|
||||
|
||||
### Why This Architecture Works
|
||||
|
||||
#### Focused Expertise
|
||||
Each household member (expert agent) receives highly specific prompts tailored to their domain. Rather than a single overly-broad prompt trying to do everything, specialized agents work within their areas of competence.
|
||||
|
||||
#### Cognitive Load Management
|
||||
By pre-filtering tools and agents, the Steward prevents Tatlock from being overwhelmed with options. This is analogous to how a real butler doesn't personally know every detail of every household operation—they know whom to ask.
|
||||
|
||||
#### Transparent Coordination
|
||||
The Steward's recommendations are visible in the thinking flow, keeping users informed about which household staff are being consulted. This transparency builds trust and understanding.
|
||||
|
||||
#### Composable Capabilities
|
||||
New expert agents can be added to the household without overwhelming the core system. The Steward learns about new staff members and includes them in recommendations when appropriate.
|
||||
|
||||
#### Model Efficiency
|
||||
Rather than requiring a single enormous context window containing all possible tools and capabilities, the system makes targeted calls with focused contexts. This is more efficient and produces better results.
|
||||
|
||||
**Unified Base Model**: All household members—the Steward, Tatlock, and expert agents—use the same base language model by default. This ensures the model stays loaded in VRAM, eliminating loading delays between calls and maximizing response speed.
|
||||
|
||||
**Specialized Models When Needed**: Individual household staff may invoke specialized models for domain-specific tasks when appropriate:
|
||||
- The Developer might use Codestral for complex code generation
|
||||
- Future visual agents might use vision-language models
|
||||
- Future audio agents might use speech-specific models
|
||||
|
||||
The decision to use a specialized model is made by the household member responsible for that domain, based on the specific requirements of their task. This balances efficiency (keeping the base model hot) with capability (accessing specialized models when they provide significant advantage).
|
||||
|
||||
### Personality and Interaction
|
||||
|
||||
While the underlying architecture is sophisticated, users interact solely with **Tatlock**, who maintains a consistent personality:
|
||||
|
||||
- **Witty but helpful**: Responses may include clever observations or light humor
|
||||
- **Competent and organized**: Always knows who to ask and how to coordinate
|
||||
- **Context-aware**: Remembers ongoing conversations and user preferences
|
||||
- **Transparent**: Explains which household staff are being consulted when relevant
|
||||
- **Professional**: Despite the wit, maintains respect and helpfulness
|
||||
|
||||
The user never directly interacts with the Steward or individual expert agents—those are internal household operations that Tatlock manages on their behalf.
|
||||
|
||||
---
|
||||
|
||||
## Document Metadata
|
||||
|
||||
**Document Type**: Architectural Philosophy (Stable)
|
||||
**Purpose**: Establish foundational patterns and guiding principles
|
||||
**Modification Policy**: Only update when deviating from or enhancing core architectural patterns
|
||||
**Version**: 1.0
|
||||
**Established**: 2025-12-06
|
||||
**Project Version**: 0.1.1
|
||||
|
||||
**Related Documents**:
|
||||
- **README.md**: User-facing documentation and usage guide
|
||||
- **AGENTS.md**: LLM agent development guidelines and technical patterns
|
||||
- **CHANGELOG.md**: Version history and implemented features
|
||||
|
||||
---
|
||||
|
||||
*All development should work towards realizing the patterns described in this document.*
|
||||
+208
@@ -0,0 +1,208 @@
|
||||
# Tatlock Implementation Roadmap
|
||||
|
||||
> **Reference**: See [philosophy.md](philosophy.md) for the target architecture and vision
|
||||
|
||||
This document tracks open/planned work. Completed phases have been removed.
|
||||
|
||||
## Current State (v2.0.5)
|
||||
|
||||
**What we have**:
|
||||
- OpenAI-compatible API (Responses API + Chat Completions)
|
||||
- Two-tier architecture (Steward → Tatlock)
|
||||
- Household staff: Tatlock (Butler), Steward, Librarian, Biographer
|
||||
- Core tools: Calculator, Date/Time, Web search (SearXNG)
|
||||
- Memory system: Qdrant (vector), Redis (session cache), multi-tenancy via ContextVar
|
||||
- Dual backend: Claude (preferred) + Ollama (fallback)
|
||||
- 439 tests with good coverage
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Expert Household Staff — Remaining Agents
|
||||
|
||||
**Goal**: Implement remaining domain-specific expert agents
|
||||
|
||||
### Planned Agents
|
||||
|
||||
1. **The Developer** (Software Development)
|
||||
- Code generation assistance
|
||||
- Debugging support
|
||||
- Documentation generation
|
||||
- Architecture guidance
|
||||
|
||||
2. **The Handyman** (System Maintenance)
|
||||
- System status queries
|
||||
- Log analysis
|
||||
- Basic troubleshooting
|
||||
- Infrastructure monitoring
|
||||
|
||||
3. **The Secretary** (Scheduling & Organization)
|
||||
- Calendar integration
|
||||
- Task management
|
||||
- Reminder system
|
||||
- Schedule conflict detection
|
||||
|
||||
4. **The Housekeeper** (Home Automation)
|
||||
- Home Assistant integration
|
||||
- Device control interface
|
||||
- Status queries
|
||||
- Automation triggers
|
||||
|
||||
### Each Agent Includes
|
||||
- Specialized prompt and personality
|
||||
- Domain-specific tools
|
||||
- MCP integration points (where applicable)
|
||||
- Integration with Butler orchestration
|
||||
|
||||
### Success Criteria
|
||||
- [ ] Each agent implemented as separate module
|
||||
- [ ] Agents callable via tool framework
|
||||
- [ ] Can invoke specialized models (e.g., Codestral for Developer)
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Persistence Layer — Database & Multi-Tenancy
|
||||
|
||||
**Goal**: Add persistent storage and multi-user support
|
||||
|
||||
### Deliverables
|
||||
|
||||
1. **PostgreSQL Integration**
|
||||
- Docker compose configuration
|
||||
- Database schema with tenant isolation
|
||||
- Alembic migrations
|
||||
- SQLAlchemy models
|
||||
|
||||
2. **Multi-Tenant Architecture**
|
||||
- Tenant identification middleware
|
||||
- Tenant-scoped database sessions
|
||||
- User authentication system
|
||||
- Per-tenant data isolation
|
||||
|
||||
3. **Core Data Models**
|
||||
- Users and tenants
|
||||
- Conversations and messages (migrate from in-memory)
|
||||
- Agent interactions log
|
||||
- System configuration and preferences
|
||||
|
||||
### Success Criteria
|
||||
- [ ] PostgreSQL container running
|
||||
- [ ] Multiple users authenticate separately
|
||||
- [ ] Each user sees only their own data
|
||||
- [ ] Conversations persist across restarts
|
||||
- [ ] Database migrations work correctly
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: MCP (Model Context Protocol) Integration
|
||||
|
||||
**Goal**: Enable rich tool integrations via MCP
|
||||
|
||||
See also [claude-integration.md](claude-integration.md) for MCP server implementation details.
|
||||
|
||||
### Deliverables
|
||||
|
||||
1. **MCP Server Framework**
|
||||
- MCP server implementation
|
||||
- Tool registration via MCP
|
||||
- Schema validation
|
||||
- Error handling
|
||||
|
||||
2. **MCP Client in Agents**
|
||||
- PydanticAI MCP integration
|
||||
- Tool discovery from MCP servers
|
||||
- Dynamic tool loading
|
||||
|
||||
3. **Initial MCP Tools**
|
||||
- File system operations
|
||||
- Database queries
|
||||
- API integrations
|
||||
- System commands
|
||||
|
||||
### Success Criteria
|
||||
- [ ] MCP server running
|
||||
- [ ] Tools exposed via MCP protocol
|
||||
- [ ] Agents can discover and use MCP tools
|
||||
- [ ] New tools addable without code changes
|
||||
- [ ] MCP tools visible in Steward recommendations
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Advanced Memory & Context — Remaining Work
|
||||
|
||||
**Goal**: Implement sophisticated context management and personalization
|
||||
|
||||
### Open Deliverables
|
||||
|
||||
1. **Context Management**
|
||||
- Smart context window trimming
|
||||
- Conversation branching
|
||||
- Topic tracking
|
||||
|
||||
2. **Personalization**
|
||||
- User preference learning
|
||||
- Interaction pattern analysis
|
||||
- Adaptive responses
|
||||
- Custom agent personalities per user
|
||||
|
||||
### Success Criteria
|
||||
- [ ] Conversations automatically embedded to Qdrant
|
||||
- [ ] Memory improves over time (learning from interactions)
|
||||
|
||||
---
|
||||
|
||||
## Phase 9: Extended Household Staff
|
||||
|
||||
**Goal**: Add specialized agents for additional domains
|
||||
|
||||
### Future Agents
|
||||
- **The Accountant** — Expense tracking, budgets, financial reports
|
||||
- **The Chef** — Meal planning, recipes, nutrition tracking
|
||||
- Others as needs emerge
|
||||
|
||||
---
|
||||
|
||||
## Phase 10: User Experience Refinement
|
||||
|
||||
**Goal**: Polish the interaction experience
|
||||
|
||||
- Personality tuning and consistency
|
||||
- Better progress indicators
|
||||
- Response time improvements
|
||||
- Streaming smoothness
|
||||
|
||||
---
|
||||
|
||||
## Phase 11: Production Hardening
|
||||
|
||||
**Goal**: Make the system production-ready for homelab deployment
|
||||
|
||||
- Complete docker-compose stack
|
||||
- Health checks and monitoring
|
||||
- Authentication hardening and rate limiting
|
||||
- Installation and troubleshooting documentation
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```
|
||||
Phase 4 (Remaining Agents)
|
||||
↓
|
||||
Phase 5 (Database/Multi-Tenancy) ← Can be deferred
|
||||
↓
|
||||
Phase 7 (MCP) → Phase 8 (Advanced Memory)
|
||||
↓
|
||||
Phase 9 (Extended Staff) → Phase 10 (UX) → Phase 11 (Production)
|
||||
```
|
||||
|
||||
**Can Be Deferred**: Phase 5 until you need persistence
|
||||
**Parallel Opportunities**: Phases 7 and 8 can overlap; 9 and 10 ongoing
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Implement The Developer agent for code assistance
|
||||
2. Add Home Assistant integration for The Housekeeper
|
||||
3. Integrate scheduling service for The Secretary
|
||||
4. MCP server for external Claude access
|
||||
@@ -0,0 +1,105 @@
|
||||
# Testing Improvements for LLM Outputs
|
||||
|
||||
## Problem
|
||||
|
||||
LLM outputs are non-deterministic. Tests checking for exact string matches fail when the LLM writes "thirty-seven" instead of "37".
|
||||
|
||||
## Proposed Solutions
|
||||
|
||||
### 1. LLM-as-Judge Pattern
|
||||
|
||||
Use a smaller/faster model to evaluate semantic correctness:
|
||||
|
||||
```python
|
||||
async def llm_judge(output: str, criteria: str) -> bool:
|
||||
"""Use LLM to evaluate if output meets criteria."""
|
||||
prompt = f"""
|
||||
Evaluate if this output is correct:
|
||||
Output: {output}
|
||||
Criteria: {criteria}
|
||||
Answer only YES or NO.
|
||||
"""
|
||||
result = await judge_model.run(prompt)
|
||||
return "YES" in result.output.upper()
|
||||
|
||||
# Usage in test:
|
||||
assert await llm_judge(
|
||||
response,
|
||||
"The answer correctly states that sqrt(144) + 25 = 37"
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Fuzzy/Regex Matching
|
||||
|
||||
For numeric answers, accept multiple representations:
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def contains_number(text: str, number: int) -> bool:
|
||||
"""Check if text contains number in any form."""
|
||||
patterns = [
|
||||
rf'\b{number}\b', # Digit form
|
||||
number_to_words(number), # Word form
|
||||
]
|
||||
return any(re.search(p, text, re.I) for p in patterns)
|
||||
|
||||
# Usage:
|
||||
assert contains_number(response, 37) # Matches "37" or "thirty-seven"
|
||||
```
|
||||
|
||||
### 3. DeepEval Framework
|
||||
|
||||
```python
|
||||
from deepeval.metrics import AnswerRelevancyMetric
|
||||
from deepeval.test_case import LLMTestCase
|
||||
|
||||
def test_calculation():
|
||||
test_case = LLMTestCase(
|
||||
input="What is sqrt(144) + 25?",
|
||||
actual_output=response,
|
||||
expected_output="37"
|
||||
)
|
||||
metric = AnswerRelevancyMetric(threshold=0.7)
|
||||
assert metric.measure(test_case)
|
||||
```
|
||||
|
||||
### 4. pytest-evals Plugin
|
||||
|
||||
Minimal pytest plugin for LLM testing with metrics collection.
|
||||
|
||||
```bash
|
||||
pip install pytest-evals
|
||||
```
|
||||
|
||||
### 5. Multiple Runs with Threshold
|
||||
|
||||
Run flaky tests multiple times and require majority pass:
|
||||
|
||||
```python
|
||||
@pytest.mark.flaky(reruns=3, reruns_delay=1)
|
||||
def test_llm_response():
|
||||
...
|
||||
```
|
||||
|
||||
Or custom:
|
||||
|
||||
```python
|
||||
@pytest.mark.parametrize("run", range(3))
|
||||
def test_llm_response(run):
|
||||
...
|
||||
# Aggregate results across runs
|
||||
```
|
||||
|
||||
## Resources
|
||||
|
||||
- [DeepEval](https://github.com/confident-ai/deepeval) - LLM evaluation framework
|
||||
- [pytest-evals](https://github.com/AlmogBaku/pytest-evals) - pytest plugin for LLM evals
|
||||
- [LLM Testing Guide 2025](https://www.confident-ai.com/blog/llm-testing-in-2024-top-methods-and-strategies)
|
||||
- [Testing LLM Applications - Langfuse](https://langfuse.com/blog/2025-10-21-testing-llm-applications)
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
1. Add fuzzy number matching helper (quick win)
|
||||
2. Evaluate DeepEval for complex output testing
|
||||
3. Consider LLM-as-judge for semantic correctness
|
||||
Reference in New Issue
Block a user