jpmschweitzer/portainer-core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: prune obsolete documentation and consolidate

Browse Source 
- Remove obsolete plans/ directory (AI orchestrator moved to separate services)
- Remove docs/sessions/ (historical session notes)
- Remove docs/reference/ (duplicated in CONTAINERS.md)
- Remove docs/architecture/ (duplicated in CONTAINERS.md)
- Remove maintenance stack and backup-procedures guide (decommissioned)
- Remove scripts/ directory (unused)
- Fold EXTERNAL_SERVICES.md into CONTAINERS.md
- Update Nextcloud to reflect PostgreSQL shared (was MariaDB)
- Clean up README.md links

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2025-12-11 16:29:06 +01:00
co-authored by Claude Opus 4.5
parent 0815f02529
commit 35077ed5ca
31 changed files with 37 additions and 10669 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTAINERS.md
+27 -48
View File
@@ -21,7 +21,6 @@
| **Netdata** | 19999 | http://192.168.86.149:19999 | LAN | No | ✅ Running |
| **Organizr** | 9999 | https://home.schweitz.net | Internet (SSO) | No | ✅ Running |
| **Watchtower** | None | N/A (background) | No | No | ✅ Running |
| **Maintenance** | None | N/A (background) | No | No | ✅ Running |
| **Open WebUI** | 82 | http://192.168.86.149:82 | LAN | No | ✅ Running |
| **Core API** | 8083 | http://192.168.86.149:8083 | LAN | No | ✅ Running (external) |
| **Scheduler** | 8090 | http://192.168.86.149:8090 | LAN | No | ✅ Running (external) |
@@ -35,15 +34,26 @@
### External Domains (SSL via Let's Encrypt)
- **home.schweitz.net** → Organizr (Protected by Authentik SSO)
- **media.schweitz.net** → Jellyfin
- **cloud.schweitz.net** → Nextcloud
- **git.schweitz.net** → Gitea
- **auth.schweitz.net** → Authentik SSO
- **api.schweitz.net** → Core API
- **code.schweitz.net** → Code-Server (host service)
- **amp.schweitz.net** → AMP Game Server
- **media.schweitz.net** → Jellyfin
- **cloud.schweitz.net** → Nextcloud
- **git.schweitz.net** → Gitea
- **auth.schweitz.net** → Authentik SSO
- **api.schweitz.net** → Core API
- **code.schweitz.net** → Code-Server (host service)
- **amp.schweitz.net** → AMP Game Server
- **tatlock.schweitz.net** → (Reserved)
### External Repositories
Services marked "(external)" have source code in separate Gitea repositories. Container images are built via Gitea Actions on release and auto-updated by Watchtower.
| Service | Repository | Image |
|---------|------------|-------|
| **Scheduler** | [scheduler](https://git.schweitz.net/jpmschweitzer/scheduler) | `git.schweitz.net/jpmschweitzer/scheduler:latest` |
| **Core API** | [core-api](https://git.schweitz.net/jpmschweitzer/core-api) | `git.schweitz.net/jpmschweitzer/core-api:latest` |
**Development workflow:** Clone repo → make changes → create Gitea release → Watchtower auto-updates container.
---
## 📊 Infrastructure Management via Core API
@@ -357,30 +367,6 @@ Watchtower automatically monitors all running containers for updated images and
---
### Maintenance Container
The maintenance container runs scheduled automation tasks including nightly Docker configuration backups with 30-day retention, disk space monitoring, log cleanup, and future expansion for health checks and system maintenance scripts. It executes cron-based jobs at 3 AM daily to archive all Docker Compose configurations, container settings, and persistent data to the backup directory on the HDD with timestamped snapshots. The container provides a centralized location for all homelab automation without cluttering the host system with multiple cron entries.
| Property | Value |
|----------|-------|
| **Image** | `alpine:latest` |
| **Container Name** | `maintenance` |
| **Access URL** | N/A (background service) |
| **External Access** | N/A |
| **Port Mapping** | None (no exposed ports) |
| **Network Mode** | Bridge |
| **Restart Policy** | `unless-stopped` |
| **Volume Mounts** | `~/docker-data:/source:ro`, `/mnt/media/backups:/backups` |
| **Command** | Runs crond with custom crontab |
| **Resource Limits** | None |
| **GPU Required** | No |
| **Dependencies** | None |
| **Schedule** | Daily at 3:00 AM (backups) |
| **Backup Retention** | 30 days |
| **Backup Size** | ~94MB per snapshot |
---
## Application Layer
### Open WebUI
@@ -496,7 +482,7 @@ Jellyfin is a GPU-accelerated media server that organizes, streams, and transcod
### Nextcloud
Nextcloud is a self-hosted cloud storage and collaboration platform providing file sync, sharing, calendar, contacts, and collaborative document editing with a web interface and mobile apps, replacing cloud services like Dropbox or Google Drive while maintaining full data sovereignty. It runs as a multi-container stack with a MariaDB database for metadata, Redis for caching and file locking, and the main PHP application container, with the application configuration stored on SSD for responsiveness while user data resides on the HDD for capacity. The service integrates behind Nginx Proxy Manager with SSL at https://cloud.schweitz.net, offering external access for file synchronization from anywhere while maintaining automated background job execution through the maintenance container's cron system.
Nextcloud is a self-hosted cloud storage and collaboration platform providing file sync, sharing, calendar, contacts, and collaborative document editing with a web interface and mobile apps. Uses shared PostgreSQL for metadata and shared Redis for caching. Application config on SSD, user data on HDD. External access via NPM at https://cloud.schweitz.net.
| Property | Value |
|----------|-------|
@@ -506,15 +492,13 @@ Nextcloud is a self-hosted cloud storage and collaboration platform providing fi
| **Access URL (Public)** | https://cloud.schweitz.net |
| **External Access** | Yes (via NPM reverse proxy with SSL) |
| **Port Mapping** | 8082:80 (HTTP) |
| **Network Mode** | Bridge (custom network: nextcloud_nextcloud-network) |
| **Network Mode** | docker-dataplane |
| **Restart Policy** | `unless-stopped` |
| **Volume Mounts** | `~/docker-data/nextcloud/config:/var/www/html` (SSD), `/mnt/media/nextcloud/data:/var/www/html/data` (HDD) |
| **Environment** | `MYSQL_HOST=nextcloud-db`, `MYSQL_DATABASE=nextcloud`, `MYSQL_USER=nextcloud`, `REDIS_HOST=nextcloud-redis`, `TZ=Europe/Amsterdam` |
| **Resource Limits** | None |
| **GPU Required** | No |
| **Dependencies** | MariaDB 10.11 (nextcloud-db), Redis Alpine (nextcloud-redis), NPM (reverse proxy), Maintenance container (cron jobs) |
| **Database** | MariaDB on SSD (~100MB) |
| **Cron Jobs** | Background tasks every 5 minutes (via maintenance container) |
| **Dependencies** | PostgreSQL Shared, Redis Shared, NPM (reverse proxy) |
| **Database** | PostgreSQL (shared) |
| **Storage Split** | Config/apps on SSD, user data on HDD |
| **Features** | File sync, calendar, contacts, document editing, photo gallery, mobile apps |
@@ -595,7 +579,6 @@ Gitea is a lightweight, self-hosted Git service providing repository hosting, is
| **Gitea** | https://git.schweitz.net | Yes | Git repository hosting |
| **Samba** | \\\\192.168.86.149 | No | Network file shares |
| **Watchtower** | N/A (background) | N/A | Auto-updates |
| **Maintenance** | N/A (background) | N/A | Automated tasks |
---
@@ -631,7 +614,6 @@ Gitea is a lightweight, self-hosted Git service providing repository hosting, is
| **Nextcloud** | `~/docker-data/nextcloud/` | `/mnt/media/nextcloud/data/` | Config: ~200MB, DB: ~100MB, User data: variable |
| **Gitea** | `~/docker-data/gitea/` | N/A | Data: ~100MB, DB: ~50MB, Repos: variable |
| **Samba** | `~/docker-data/samba/` | Mounts: `/mnt/media/` (shares) | Config: ~5MB |
| **Maintenance** | N/A | `/mnt/media/backups/` | ~94MB per backup |
**SSD Usage (docker-data):** ~6-11GB (configs, caches, databases)
**HDD Usage (/mnt/media):** ~2.1TB / 3.6TB (58% used)
@@ -644,7 +626,7 @@ Gitea is a lightweight, self-hosted Git service providing repository hosting, is
| Network Name | Containers | Purpose |
|--------------|------------|---------|
| **docker-dataplane** | Ollama, Open WebUI, Core API, Qdrant, Uptime Kuma, PostgreSQL Shared, Redis Shared, Headscale, Nextcloud, Gitea, Samba, Watchtower, Maintenance, Organizr, Netdata | Unified service mesh for all containerized applications |
| **docker-dataplane** | Ollama, Open WebUI, Core API, Qdrant, Uptime Kuma, PostgreSQL Shared, Redis Shared, Headscale, Nextcloud, Gitea, Samba, Watchtower, Organizr, Netdata | Unified service mesh for all containerized applications |
| **host** | Portainer, NPM | Direct host port access for infrastructure management |
**Benefits of Consolidation**:
@@ -673,17 +655,14 @@ redis-cli -h redis-shared # Redis connection
---
## Maintenance Schedule
## Automated Tasks
| Service | Task | Frequency | Time |
|---------|------|-----------|------|
| **Watchtower** | Container updates | Daily | 4:00 AM |
| **Maintenance** | Config backups | Daily | 3:00 AM |
| **Maintenance** | Nextcloud background jobs | Every 5 minutes | Continuous |
| **Maintenance** | Log cleanup | Weekly | Sunday 3:30 AM |
| **Docker** | Image pruning | Monthly | 1st of month |
| **Scheduler** | Config backups | Daily | 3:05 AM |
| **Scheduler** | Doc sync | Monthly | 11th/12th |
---
*Last Updated: 2025-11-16*
*System: tower-of-joy (tower-of-joy v0.5.0-optimization)*
*Last Updated: 2025-12-11*
PLANS.md
-159
View File
@@ -1,159 +0,0 @@
# Implementation Plans
This document tracks all implementation plans across the portainer-core project.
## Active Plans
Current implementation work in progress:
### AI Orchestrator Enhancement
**Location**: [plans/active/ai-orchestrator-plan.md](plans/active/ai-orchestrator-plan.md)
**Status**: ✅ Phase 4 Complete - ADK Migration Successful
**Phases**:
- ✅ Phase 1: OpenAI-Compatible API (Completed 2025-11-13)
- ✅ Phase 2: Memory Systems (Completed 2025-11-23)
- ✅ Phase 3: Research Capabilities (Completed 2025-11-24)
- ✅ Phase 4: Framework Migration - LangChain → Google ADK (Completed 2025-11-26)
- 📋 Phase 5: Multi-Agent Patterns (Future)
- 📋 Phase 6: Production Hardening & RAG Optimization (Future)
**Framework Migration Completed (2025-11-26)** ✅:
- ✅ Migrated from LangChain/LangGraph to Google ADK 1.3.0
- ✅ Integrated LiteLLM 1.80.5 for Ollama compatibility
- ✅ Converted all 9 tools to ADK async generator format
- ✅ Upgraded model: mistral:7b → gemma3:12b
- ✅ Optimized system prompt: v7_adk_best_practice
- ✅ Enhanced agent health monitoring
- ✅ Production testing and validation
**Migration Benefits Achieved**:
- Improved tool calling reliability with Ollama models
- Better streaming support with ADK event system
- Model flexibility (Gemma, Mistral, Qwen families supported)
- Cleaner, more maintainable architecture
- Production-ready health monitoring
**Current Implementation**:
- Framework: Google ADK 1.3.0 with LiteLLM
- Model: gemma3:12b (~8GB VRAM)
- Tools: 9 total (7 infrastructure + 2 research)
- Performance: Simple queries ~0.3-1s, Research ~4-7s
### Memory Architecture
**Location**: [plans/completed/phase2-memory-system-complete.md](plans/completed/phase2-memory-system-complete.md)
**Status**: ✅ Completed 2025-11-23
**Description**: 3-tier memory system (buffer, Qdrant persistent + semantic) with multi-tenancy
### Security Implementation
**Location**: [plans/active/security-implementation-plan.md](plans/active/security-implementation-plan.md)
**Status**: 📋 Planning Phase
**Description**: Google OAuth SSO via Authentik for external service access
---
## Completed Plans
Historical implementation plans that have been finished:
### Infrastructure Deployment (Phases 1-4)
**Location**: [plans/completed/infrastructure-deployment-plan.md](plans/completed/infrastructure-deployment-plan.md)
**Completed**: November 2025
**Phases**:
- ✅ Phase 1: Foundation (Portainer, NPM, Ollama)
- ✅ Phase 2: Networking (Headscale mesh VPN)
- ✅ Phase 3: Monitoring (Uptime Kuma, Netdata, Heimdall)
- ✅ Phase 4: Optimization (Watchtower, Duplicati)
### AI Orchestrator Phase 1
**Location**: [plans/completed/ai-orchestrator-phase1-guide.md](plans/completed/ai-orchestrator-phase1-guide.md)
**Completed**: November 2025
**Deliverables**: OpenAI-compatible API with model routing, streaming, function calling
### AI Orchestrator Phase 1 Testing
**Location**: [plans/completed/ai-orchestrator-phase1-tests.md](plans/completed/ai-orchestrator-phase1-tests.md)
**Results**: 10/10 tests passed, zero issues found
### AI Orchestrator Phase 2 (Memory System)
**Location**: [plans/completed/phase2-memory-system-complete.md](plans/completed/phase2-memory-system-complete.md)
**Completed**: 2025-11-23
**Deliverables**: 3-tier memory (buffer + Qdrant), multi-tenancy, auto-consolidation
### AI Orchestrator Phase 3 (Research Capabilities)
**Location**: [plans/completed/phase3-multi-agent-workflows-complete.md](plans/completed/phase3-multi-agent-workflows-complete.md)
**Completed**: 2025-11-24
**Deliverables**: Web search (DuckDuckGo), content scraping, research detection, 100% test success
### AI Orchestrator Phase 4 (Framework Migration)
**Location**: [MIGRATION_PLAN_LANGCHAIN_TO_ADK.md](MIGRATION_PLAN_LANGCHAIN_TO_ADK.md)
**Completed**: 2025-11-26
**Deliverables**: Google ADK 1.3.0 with LiteLLM, 9 tools migrated, gemma3:12b model, improved reliability
### Architecture Research
**Location**: [plans/completed/architecture-research.md](plans/completed/architecture-research.md)
**Completed**: October 2025
**Decision**: Portainer + Docker Compose for container orchestration
### Mesh Networking Strategy
**Location**: [plans/completed/mesh-networking-strategy.md](plans/completed/mesh-networking-strategy.md)
**Completed**: November 2025
**Solution**: Headscale (self-hosted Tailscale) for secure mesh VPN
### Dashboard Consolidation Strategy
**Location**: [plans/completed/dashboard-strategy.md](plans/completed/dashboard-strategy.md)
**Completed**: November 2025
**Solution**: Organizr with custom service control widgets
---
## Plan Management
### Creating New Plans
1. Create plan in `plans/active/` directory
2. Add entry to "Active Plans" section above
3. Update STATUS.md with phase tracking
4. Link from relevant documentation
### Completing Plans
1. Mark all phases as ✅ in the plan document
2. Move from `plans/active/` to `plans/completed/`
3. Update this file (move to "Completed Plans" section)
4. Update STATUS.md
5. Update CHANGELOG.md with release notes
### Plan Template
```markdown
# [Feature Name] Implementation Plan
## Overview
Brief description of the feature/improvement.
## Motivation
Why this change is needed.
## Phases
### Phase 1: [Name]
**Status**: 📋 Planned / 🔄 In Progress / ✅ Completed
**Duration**: Estimated effort
**Deliverables**:
- [ ] Task 1
- [ ] Task 2
## Success Criteria
How to determine if implementation is complete.
## Testing Strategy
How the feature will be validated.
```
---
## Quick Links
- [Project Status](STATUS.md) - Current phase and progress tracking
- [Documentation Index](README.md) - All project documentation
- [Active Plans](plans/active/) - Current implementation work
- [Completed Plans](plans/completed/) - Historical implementations
README.md
+10 -35
View File
@@ -1,48 +1,23 @@
# portainer-core
> Self-hosted home server infrastructure with GPU-accelerated ML, AI orchestration, media streaming, and secure remote access
> Self-hosted home server infrastructure with GPU-accelerated ML, media streaming, and secure remote access
**Main Dashboard:** https://home.schweitz.net (Organizr)
## Quick Links
### Getting Started
- [System Specifications](docs/reference/SYSTEM.md) - Hardware and software details
- [Container Reference](docs/reference/CONTAINERS.md) - All deployed services
- [Current Status](STATUS.md) - Implementation progress and phase tracking
### Documentation
### Implementation Plans
- [Implementation Plans](PLANS.md) - Master plan tracker
- [Active Plans](plans/active/) - Current development work
- [Completed Plans](plans/completed/) - Historical implementations
### Documentation Index
#### Architecture & Design
- [Shared Infrastructure Architecture](docs/architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md) - PostgreSQL/Redis shared infrastructure
#### Operational Guides
- [Backup Procedures](docs/guides/backup-procedures.md) - Backup strategies and procedures
- [Code-Server Setup](docs/guides/code-server-setup.md) - Browser-based IDE configuration
- [Connect Devices Guide](docs/guides/connect-devices-guide.md) - Headscale VPN setup
- [GPU Docker Configuration](docs/guides/gpu-docker-config.md) - NVIDIA GPU passthrough
- [Headscale Setup](docs/guides/headscale-setup.md) - Mesh VPN deployment
- [NPM Logging Guide](docs/guides/npm-logging-guide.md) - Nginx Proxy Manager logging
#### Services
- [Organizr Widgets](docs/services/organizr-widgets.md) - Service control dashboard
- [External Services](docs/EXTERNAL_SERVICES.md) - Services in separate repositories
#### Reference
- [Stacks Reference](docs/reference/stacks.md) - All Docker Compose stacks
- [Scripts Reference](docs/reference/scripts.md) - Maintenance automation
- [Automation Reference](docs/reference/AUTOMATION.md) - Portainer REST API usage
- [Container Reference](docs/reference/CONTAINERS.md) - Complete container profiles
- [System Reference](docs/reference/SYSTEM.md) - Hardware specifications
- [Container Reference](CONTAINERS.md) - All services, ports, configuration
- [Changelog](CHANGELOG.md) - Version history
- [Agent Guidelines](AGENTS.md) - For LLM coding agents
### For AI Agents
- [Agent Guidelines](AGENTS.md) - **REQUIRED READING** for all LLM coding agents
#### Guides
- [GPU Docker Configuration](docs/guides/gpu-docker-config.md)
- [Headscale VPN Setup](docs/guides/headscale-setup.md)
- [Connect Devices to VPN](docs/guides/connect-devices-guide.md)
- [Code-Server Setup](docs/guides/code-server-setup.md)
- [NPM Logging](docs/guides/npm-logging-guide.md)
## Architecture Overview
STATUS.md
-207
View File
@@ -1,207 +0,0 @@
# Project Status
> **Last Updated:** 2025-12-03
> **Version:** 0.11.0-pydantic-ai
## Current Phase
**Active Work:** AI Infrastructure Optimization & System Hardening
**Status:** ✅ **STABLE** - PydanticAI Implementation Complete, All Systems Operational
See [PLANS.md](PLANS.md) for complete implementation roadmap and [CHANGELOG.md](CHANGELOG.md) for version history.
## In Progress
### Priority 1: Security & SSO Implementation (Authentik)
- [x] **Milestone 1: Authentik Deployment**
- [x] Deploy Authentik server and worker containers
- [x] Configure shared PostgreSQL database (authentik_user, authentik database)
- [x] Configure shared Redis (DB 0)
- [x] Fix health checks (Python urllib instead of wget/curl)
- [x] Create NPM proxy host for auth.schweitz.net
- [x] Generate admin recovery key and set password
- [x] Memory optimization: 563MB total (80-90% reduction vs previous attempt)
- [x] **Milestone 2: Google OAuth Integration**
- [x] Create Google OAuth credentials (Client ID/Secret)
- [x] Configure Authentik Google source via API
- [x] Configure identification stage to show social login
- [x] Test Google OAuth login (successful)
- [x] Verify user creation (jpmschweitzer@gmail.com - external type)
- [x] **Milestone 3: Forward Auth for Organizr** ✅ COMPLETE (2025-11-21)
- [x] Create Authentik Proxy Provider (Organizr Proxy) via API
- [x] Create Authentik Application (Organizr) via API
- [x] ~~Assign provider to embedded outpost~~ (embedded outpost failed)
- [x] **Deploy standalone outpost container** (authentik-proxy on port 9443)
- [x] Configure Redis connection for standalone outpost
- [x] Verify outpost endpoints operational
- [x] **Configure NPM forward auth for home.schweitz.net**
- [x] Test SSO access to Organizr (Google OAuth login working)
- [x] Verify no redirect loops
- [x] Fix Organizr auto-login (moved headers to location / block)
**Resolution:** Embedded outpost has version-specific issues in 2024.8.4. Deployed standalone `authentik-proxy` container successfully. Forward auth fully operational with Organizr auto-login working.
**Standalone Outpost Details:**
- Container: `authentik-proxy` (port 9445:9443)
- Status: ✅ Healthy (websocket connected, ping endpoint responding)
- Memory: ~150MB
- Provider: Organizr Proxy (forward_single mode)
- Token: `9blMGz71CFMJszs7AedQefgydpTnwvybjmMn0AlYilIKBV5LIq7snqnCodwX`
**NPM Configuration:**
- Applied to: home.schweitz.net (Organizr) ONLY
- Forward auth: https://localhost:9445/outpost.goauthentik.io (NPM on host network)
- WebSocket support: Enabled
- Headers: X-authentik-username, X-authentik-email, X-authentik-groups, X-authentik-name, X-authentik-uid
- Status: ✅ Fully operational, tested in incognito
**Critical Fix:** Authentication headers must be set inside `location /` block, not at server level, for proper forwarding to backend applications.
### Priority 2: Core-API Refactoring & Infrastructure Management ✅ COMPLETE
- [x] **Code Cleanup:** Restructure Core API into function-specific controller files
- [x] Create `/controllers` directory structure
- [x] Create `/clients` directory structure
- [x] Create `base.py` controller base class
- [x] Add infrastructure settings to `config.py`
- [x] Create credentials management system
- [x] Update `main.py` routing to include infrastructure controller
- [x] Separate AI Orchestrator logic into `ai_controller.py`
- [x] Extract webscraper to `tools_controller.py`
- [x] Create `health_controller.py` for monitoring endpoints
- [x] **Infrastructure Management Controller:** Build automation API for service management
- [x] Portainer Integration (HTTP client with access token)
- [x] NPM Integration (HTTP client with JWT bearer token + auto-refresh)
- [x] Read/List Endpoints (all implemented & tested)
- [x] Write Endpoints (POST/PUT/DELETE all implemented & tested)
- [x] Portainer API Token generated programmatically
- [ ] Uptime Kuma Integration (deferred - complex Socket.IO)
- [ ] Replace ad-hoc shell scripts in `/stacks` with API endpoints
- [ ] Add CLI wrapper for common operations
### Priority 3: AI Orchestrator Phase 2 (Memory Systems) ✅ COMPLETE
- [x] Implement Tier 1: ConversationBufferMemory (in-memory, last 10 turns)
- [x] Implement Tier 2/3: Unified Qdrant storage (persistent + semantic search)
- [x] Create Qdrant collection (core_api_conversations with 768d nomic-embed-text)
- [x] Implement auto-consolidation service (triggers at 10 turns)
- [x] Add memory persistence across container restarts
- [x] Implement dual-retrieval (buffer + Qdrant)
- [x] **Phase 2.5: Multi-Tenancy** (user_id isolation with default "llm-testuser")
**Implementation Details:**
- **Tier 1 (Buffer):** In-memory storage for last 10 turns (< 1ms access)
- **Tier 2/3 (Qdrant):** Unified persistent storage + semantic search (768d embeddings)
- **Auto-Consolidation:** Automatically moves buffer → Qdrant at 10 turns
- **Multi-Tenancy:** Single collection with user_id filtering (default: "llm-testuser")
- **Embedding Model:** nomic-embed-text (768 dimensions, via Ollama)
- **Memory Retrieval:** Dual-check buffer + Qdrant for cross-restart persistence
- **Status:** 32 points stored, tested with multiple users, recall working after restarts
### Priority 4: AI Orchestrator - PydanticAI Implementation ✅ COMPLETE (2025-12-03)
- [x] **Phase 3:** Research Capabilities (web search, web scraping) - COMPLETE
- [x] **Framework Cleanup:** Remove obsolete frameworks (ADK, LangChain, LangGraph, OllamaNative)
- [x] **PydanticAI Agent:** Primary agent with tool calling and memory support
- [x] **Tool System:** Local tools + OpenAPI discovery from core-api
- [x] **Memory Integration:** 3-tier system with Qdrant vector storage
- [x] **Code Cleanup:** Removed all diagnostic files and obsolete implementations
- [x] **Documentation Update:** Updated all docs to reflect PydanticAI architecture
- [x] Production testing and validation
**Current Implementation (as of 2025-12-03):**
- **Framework:** PydanticAI with native Ollama SDK
- **Model:** mistral-nemo:latest (optimized for tool calling)
- **System Prompt:** Tatlock persona with infrastructure tool guidance
- **Agent:** PydanticAgent (primary) + SimpleLiteLLMAgent (fallback)
- **Tool Discovery:** Local tools + OpenAPI auto-discovery from core-api
- **Architecture:** Clean PydanticAI-only implementation
**Tools Available:**
- **Core Tools (6):** get_current_time, get_current_date, calculate_date_difference, add_days_to_date, calculate, web_search
- **Infrastructure Tools (28+):** Discovered via OpenAPI from core-api (services, DNS, domains, monitoring, etc.)
**Benefits:**
- ✅ Type-safe tool definitions with Pydantic validation
- ✅ Native Ollama SDK integration (no compatibility layers)
- ✅ Dynamic tool discovery via OpenAPI spec
- ✅ Clean architecture with single framework
- ✅ Better maintainability (no framework confusion)
**Performance Metrics:**
- **Simple queries:** ~1-3s response time
- **Tool-using queries:** ~2-8s response time
- **Memory:** Persistent across restarts via Qdrant
- **VRAM usage:** ~4-6GB with mistral-nemo:latest
**Optional Future Enhancements (deferred):**
- Multi-agent routing patterns (Phase 4+)
- Code specialist agent with codestral (Phase 4+)
- Time-based memory consolidation
- User filtering in Qdrant queries
- User management API endpoints
## Current Blockers
**None** - SSO implementation complete for critical services. Remaining service rollout deferred in favor of other priorities.
## Key Metrics
| Metric | Target | Current | Status |
|--------|--------|---------|--------|
| **Containers Running** | 15+ | 22 | 🟢 All Services Operational |
| **GPU Accessible** | Yes | Yes | 🟢 Working (RTX 2080 Ti) |
| **Storage Used** | <80% | 58% HDD (3.6TB/3.7TB) | 🟢 Healthy |
| **Services Accessible** | All | 21/21 | 🟢 Complete |
| **Remote Access** | Working | Ready | 🟢 Headscale + NPM |
| **Firewall Active** | Yes | Yes | 🟢 UFW Configured |
| **Backups Configured** | Yes | Yes | 🟢 Daily @ 3 AM |
| **AI Orchestrator** | Phase 6 | Phase 4 ✅ | 🟢 PydanticAI Complete |
| **SSO (Authentik)** | Phase 5 | Core Complete ✅ | 🟢 Organizr + Core API Protected |
## Quick Reference
### Documentation
- [Implementation Plans](PLANS.md) - Master plan tracker and roadmap
- [Changelog](CHANGELOG.md) - Version history
- [Container Reference](docs/reference/CONTAINERS.md) - All deployed services
- [System Specifications](docs/reference/SYSTEM.md) - Hardware and software details
- [Agent Guidelines](AGENTS.md) - Development conventions
### Key Paths
- **SSD configs:** `/home/jpmschweitzer/docker-data/`
- **HDD content:** `/mnt/media/`
- **Stacks:** Managed in Portainer web UI
- **Scripts:** `/mnt/media/Projects/portainer-core/scripts/`
### Active Services & URLs
**Infrastructure:**
- **Portainer:** http://192.168.86.149:8080 (container management)
- **Nginx Proxy Manager:** http://192.168.86.149:8000 (reverse proxy admin)
- **Authentik:** https://auth.schweitz.net (SSO identity provider - Google OAuth enabled)
- **Ollama:** http://192.168.86.149:11434 (ML models API)
**Networking:**
- **Headscale:** http://192.168.86.149:8085 (mesh VPN control)
**Monitoring:**
- **Uptime Kuma:** http://192.168.86.149:3001 (service monitoring)
- **Netdata:** http://192.168.86.149:19999 (system metrics)
- **Organizr:** http://192.168.86.149:8084 OR https://home.schweitz.net (unified dashboard)
**Applications:**
- **Open WebUI:** http://192.168.86.149:8081 (LLM chat interface)
- **Core API:** http://192.168.86.149:8083 (infrastructure management & AI orchestration)
- **Jellyfin:** http://192.168.86.149:8096 OR https://media.schweitz.net (GPU media server)
- **Nextcloud:** http://192.168.86.149:8082 OR https://cloud.schweitz.net (cloud storage)
- **Gitea:** http://192.168.86.149:3002 OR https://git.schweitz.net (Git hosting, SSH: 2222)
- **Samba:** \\\\192.168.86.149 or \\\\tower-of-joy (file shares: Media, Downloads, Backups)
**Background Services:**
- **Watchtower:** Automatic updates daily @ 4 AM
- **Maintenance:** Automated backups daily @ 3 AM
---
*For detailed implementation history and completed work, see [CHANGELOG.md](CHANGELOG.md)*
docs/EXTERNAL_SERVICES.md
-83
View File
@@ -1,83 +0,0 @@
# External Services
Services that have been extracted to their own repositories but are still part of the overall infrastructure.
---
## Scheduler
**Repository:** https://git.schweitz.net/jpmschweitzer/scheduler
**Container Image:** `git.schweitz.net/jpmschweitzer/scheduler:latest`
**Stack File:** `stacks/scheduler.yml` (remains in portainer-core)
### Overview
The Scheduler service handles system-wide maintenance orchestration including:
- Docker config backups
- Documentation mirroring
- Task automation via REST API
- Integration with Library Desk for consolidation jobs
### Deployment
- Container image built via Gitea Actions on release
- Stack file in portainer-core defines volumes, environment, and network
- Watchtower monitors for image updates
### Development
To make changes:
1. Clone: `git clone gitea:jpmschweitzer/scheduler.git`
2. Make changes
3. Create a release in Gitea to trigger build
4. Watchtower will auto-update the running container
### API
- Health: `http://scheduler:8090/health`
- Tasks: `http://scheduler:8090/tasks` (requires API key)
- Docs: `http://scheduler:8090/docs`
---
## Core-API
**Repository:** https://git.schweitz.net/jpmschweitzer/core-api
**Container Image:** `git.schweitz.net/jpmschweitzer/core-api:latest`
**Stack File:** `stacks/core-api.yml` (remains in portainer-core)
### Overview
Core-API provides infrastructure orchestration and OpenAI-compatible API endpoints:
- Infrastructure management (Portainer, NPM, Uptime Kuma integration)
- Web scraping tools
- AI metrics proxy
- Service health monitoring
### Deployment
- Container image built via Gitea Actions on release
- Stack file in portainer-core defines volumes, environment, and network
- Watchtower monitors for image updates
### Development
To make changes:
1. Clone: `git clone gitea:jpmschweitzer/core-api.git`
2. Make changes
3. Create a release in Gitea to trigger build
4. Watchtower will auto-update the running container
### API
- Health: `http://core-api:8083/health/full`
- Docs: `http://core-api:8083/docs`
---
## Tatlock
**Repository:** https://git.schweitz.net/jpmschweitzer/tatlock
**Location:** `/home/jpmschweitzer/Projects/tatlock`
### Overview
Tatlock is a separate project maintained in its own repository.
---
## Future Migrations
The following services are planned for extraction:
- **library-desk** - Knowledge management and consolidation service
docs/architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md
-294
View File
@@ -1,294 +0,0 @@
# Shared Infrastructure Architecture
**Purpose:** Centralized PostgreSQL and Redis services for all homelab stacks
**Benefits:** Resource efficiency, easier maintenance, unified backups, centralized monitoring
---
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────┐
│ Application Stacks │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Authentik │ │ Gitea │ │ Organizr │ │ Future │ │
│ │ │ │ │ │ │ │ Stack │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │ │
└───────┼─────────────┼──────────────┼──────────────┼─────────┘
│ │ │ │
└─────────────┴──────────────┴──────────────┘
│
┌─────────────▼──────────────────────────────┐
│ Unified Data Plane Network │
│ (docker-dataplane) │
└─────────────┬──────────────────────────────┘
│
┌─────────────┴──────────────┐
│ │
┌────▼──────┐ ┌────────▼────┐
│PostgreSQL │ │ Redis │
│ Shared │ │ Shared │
│ │ │ │
│ Databases:│ │ DB 0: Cache │
│ - auth │ │ DB 1: Auth │
│ - gitea │ │ DB 2: Gitea │
│ - organizr│ │ DB 3-15: .. │
│ - future │ │ │
└───────────┘ └─────────────┘
```
---
## Design Principles
### 1. **Database Isolation**
- Each application gets its own PostgreSQL database within the shared instance
- Each application gets its own Redis database number (0-15)
- Separate credentials per application for security
### 2. **Network Architecture**
- **Unified network:** `docker-dataplane` (external, bridge)
- All application containers connect to this single network
- Simplified connectivity: services discover each other by container name
- Replaces per-stack networks (ai-dataplane, nextcloud-network, etc.)
### 3. **Resource Allocation**
- PostgreSQL: No hard limits (homelab resource availability)
- Redis: No hard limits (lightweight Alpine image)
- Shared instances more efficient than per-stack deployments
### 4. **Backup Strategy**
- Single PostgreSQL backup covers all databases
- Automated pg_dumpall for disaster recovery
- Redis persistence: AOF + RDB snapshots
### 5. **Security Model**
- Each app has dedicated PostgreSQL user with access only to its database
- Redis AUTH with per-database passwords (optional)
- Network-level isolation via Docker networks
---
## Database Allocation Plan
### PostgreSQL Databases
| Database Name | Application | User | Purpose |
|---------------|-------------|------|---------|
| `authentik` | Authentik | `authentik_user` | User/group/policy storage |
| `gitea` | Gitea | `gitea_user` | Git repos, users, issues |
| `organizr` | Organizr | `organizr_user` | Dashboard configuration and user data |
| `future_app1` | TBD | `app1_user` | Reserved |
| `future_app2` | TBD | `app2_user` | Reserved |
**Note:** Existing services stay as-is:
- Nextcloud: MariaDB (existing, not migrated)
- Others can migrate over time if beneficial
### Redis Database Numbers
| DB# | Application | Purpose |
|-----|-------------|---------|
| 0 | Authentik | Sessions, cache, message queue |
| 1 | Available | Reserved for future applications |
| 2 | Available | Reserved for future applications |
| 3-15 | Available | Reserved for future applications |
**Note:** Each application uses a dedicated DB number to prevent key collisions while sharing the same Redis instance.
---
## Connection Configuration
### PostgreSQL Connection Strings
**From Docker containers:**
```
Host: postgres-shared
Port: 5432
Database: authentik
User: authentik_user
Password: <app-specific-password>
```
**From host:**
```
Host: localhost
Port: 5432
Database: authentik
User: authentik_user
Password: <app-specific-password>
```
### Redis Connection Strings
**From Docker containers:**
```
redis://redis-shared:6379/0 (for Authentik, DB 0)
redis://redis-shared:6379/1 (for future apps, DB 1)
redis://redis-shared:6379/2 (for future apps, DB 2)
```
**From host:**
```
redis://localhost:6379/0
```
---
## Migration Strategy
### Phase 1: Deploy Shared Infrastructure ✅ **COMPLETE**
1. ✅ Deployed `postgres-shared.yml` and `redis-shared.yml` via Portainer
2. ✅ Verified PostgreSQL 17 and Redis 7 running on docker-dataplane
3. ✅ Created initial databases and users (authentik, gitea)
4. ✅ Both services monitored via Uptime Kuma
### Phase 2: New Services (Authentik) 🚧 **IN PROGRESS**
1. ⏳ Deploy Authentik pointing to shared services
2. ⏳ Test thoroughly
3. ⏳ Validate no performance degradation
### Phase 3: Network Consolidation ✅ **COMPLETE**
1. ✅ All services migrated to docker-dataplane network
2. ✅ Removed 7 obsolete Docker networks
3. ✅ 18 containers on unified network for service discovery
### Phase 4: Migrate Existing Services (Optional)
1. **Gitea**: Already uses PostgreSQL
- Export existing database
- Create gitea database in shared PostgreSQL
- Import data
- Update Gitea stack to use shared PostgreSQL
- Remove old gitea-db container
2. **Other services**: Evaluate case-by-case
- Nextcloud: Keep MariaDB (complex migration, low benefit)
- Future services: Use shared from day 1
---
## Advantages
✅ **Resource Efficiency**
- One PostgreSQL instance: ~1GB RAM vs ~300MB per instance
- Saves ~700MB RAM per additional service using PostgreSQL
✅ **Operational Simplicity**
- Single backup process for all PostgreSQL databases
- Centralized monitoring and health checks
- Easier version upgrades (upgrade once, affects all)
✅ **Performance**
- Shared connection pooling
- Better resource utilization
- Optimized caching with shared Redis
✅ **Scalability**
- Add new applications without deploying new database instances
- Up to 15 Redis databases (more than enough for homelab)
---
## Disadvantages & Mitigations
⚠️ **Single Point of Failure**
- **Mitigation:** Health checks, automated restarts, regular backups
- **Acceptable for homelab:** VPN access ensures admin can fix issues
⚠️ **Resource Contention**
- **Mitigation:** PostgreSQL connection limits per database
- **Mitigation:** Redis max memory policy (LRU eviction)
- **Monitoring:** Track per-database usage
⚠️ **Version Lock-In**
- **Mitigation:** Use latest stable PostgreSQL version (17)
- **Mitigation:** Test upgrades in staging before production deployment
---
## Monitoring & Maintenance
### Health Checks
- PostgreSQL: `pg_isready` every 30s
- Redis: `redis-cli ping` every 30s
- Application connectivity tests
### Uptime Kuma Integration ✅ **DEPLOYED**
Both shared services are monitored via Uptime Kuma with automatic monitor creation through the Core API:
**PostgreSQL Monitor** (ID 20):
```bash
curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
-H "Content-Type: application/json" \
-d '{
"type": "postgres",
"name": "PostgreSQL Shared",
"interval": 60,
"retryInterval": 60,
"maxretries": 3,
"notificationIDList": [],
"accepted_statuscodes": ["200-299"],
"databaseConnectionString": "postgres://postgres:<url-encoded-password>@postgres-shared:5432/postgres"
}'
```
**Redis Monitor** (ID 18):
```bash
curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
-H "Content-Type: application/json" \
-d '{
"type": "port",
"name": "Redis Shared - Port Check",
"hostname": "redis-shared",
"port": 6379,
"interval": 60,
"retryInterval": 60,
"maxretries": 3,
"notificationIDList": [],
"accepted_statuscodes": ["200-299"]
}'
```
**Note:** When monitoring PostgreSQL with passwords containing special characters, URL-encode them (`/` → `%2F`, `=` → `%3D`).
### Backup Schedule
- **PostgreSQL:** Manual pg_dump to `/backups/` volume (automated backups pending)
- **Redis:** AOF persistence (real-time) enabled via `--appendonly yes`
### Performance Monitoring
- Query: `SELECT datname, numbackends FROM pg_stat_database;` (active connections)
- Redis: `INFO stats` (keyspace usage per database)
- Uptime Kuma dashboard: Real-time availability tracking
### Upgrade Path
1. Backup all databases
2. Test upgrade with docker-compose override
3. Deploy new version
4. Verify all applications connect successfully
5. Rollback if issues detected
---
## Implementation Status
1. ✅ Review architecture design
2. ✅ Create `postgres-shared.yml` and `redis-shared.yml` stacks
3. ✅ Deploy shared PostgreSQL 17 and Redis 7 via Portainer
4. ✅ Create initial databases (authentik, gitea)
5. ✅ Consolidate all services to docker-dataplane network
6. ✅ Implement Uptime Kuma monitoring via Core API
7. ✅ Document connection patterns and deployment procedures
8. ⏳ Update `authentik.yml` to use shared services (pending)
9. ⏳ Test Authentik with shared infrastructure (pending)
---
## Future Enhancements
- **PostgreSQL Read Replicas** (if needed for heavy read workloads)
- **Redis Sentinel** (high availability, probably overkill for homelab)
- **PgBouncer** (connection pooling if >100 connections needed)
- **Prometheus + Grafana** (metrics visualization)
docs/guides/backup-procedures.md
-219
View File
@@ -1,219 +0,0 @@
# Backup & Restore Procedures
## Overview
The `maintenance` container runs scheduled backup tasks using cron. It's a simple, reliable, "set and forget" solution.
**Current Backups:**
- **Docker Configs:** Daily at 3 AM
- **Retention:** 30 days
- **Size:** ~94 MB per backup
- **Location:** `/mnt/media/backups/docker-configs/`
**What's Backed Up:**
- ✅ All Docker container configurations
- ✅ Nginx Proxy Manager configs & SSL certificates
- ✅ Headscale database & config
- ✅ All dashboard settings (Heimdall, Organizr, Uptime Kuma)
- ✅ All service configs
- ❌ Ollama models (re-downloadable)
- ❌ Cache files
- ❌ Log files
## Automated Backups
**Schedule:** Daily at 3:00 AM (configured in crontab)
**View Backup Logs:**
```bash
# Real-time logs
docker logs -f maintenance
# Backup script logs
cat ~/docker-data/maintenance/logs/backup-configs.log
```
**List Existing Backups:**
```bash
ls -lh /mnt/media/backups/docker-configs/
```
## Manual Backup
Run a backup anytime:
```bash
docker exec maintenance /scripts/backup-configs.sh
```
## Restore from Backup
### Full Restore
1. **Stop all containers:**
```bash
docker stop $(docker ps -aq)
```
2. **Backup current state (just in case):**
```bash
mv ~/docker-data ~/docker-data.old
```
3. **Extract backup:**
```bash
cd ~
tar -xzf /mnt/media/backups/docker-configs/docker-configs-YYYYMMDD-HHMMSS.tar.gz
```
4. **Restart containers:**
```bash
docker start $(docker ps -aq)
```
5. **Verify services:**
```bash
docker ps
```
### Selective Restore (Single Service)
Restore only one service's config (example: Headscale):
```bash
# Extract only headscale directory
tar -xzf /mnt/media/backups/docker-configs/docker-configs-20251111-221349.tar.gz \
--strip-components=2 \
-C ~/docker-data/ \
docker-data/headscale
# Restart the service
docker restart headscale
```
## Adding New Maintenance Tasks
The maintenance container can run any scheduled task, not just backups.
### 1. Create New Script
```bash
# Create script file
nano ~/docker-data/maintenance/scripts/my-task.sh
# Make it executable
chmod +x ~/docker-data/maintenance/scripts/my-task.sh
```
### 2. Add to Crontab
```bash
# Edit crontab
nano ~/docker-data/maintenance/crontab
# Add your schedule (example: every Sunday at 4 AM)
# 0 4 * * 0 /scripts/my-task.sh
```
### 3. Restart Container
```bash
docker restart maintenance
```
### Examples of Future Tasks
- **Weekly cleanup:** Remove old Docker images
- **Health checks:** Verify all services are responding
- **Update checks:** Notify when container updates available
- **Database optimization:** Compact/optimize databases
- **SSL renewal checks:** Verify certificates are valid
## Testing Backup Integrity
Periodically test that backups can be restored:
```bash
# Create test directory
mkdir -p /tmp/backup-test
# Extract backup
tar -xzf /mnt/media/backups/docker-configs/docker-configs-LATEST.tar.gz \
-C /tmp/backup-test
# Verify contents
ls -la /tmp/backup-test/docker-data/
# Clean up
rm -rf /tmp/backup-test
```
## Troubleshooting
### Backup Not Running
**Check if container is running:**
```bash
docker ps | grep maintenance
```
**Check cron logs:**
```bash
docker logs maintenance
```
**Manually run backup to test:**
```bash
docker exec maintenance /scripts/backup-configs.sh
```
### Backup Taking Too Long
- Check if exclusions are working (Ollama models should be excluded)
- Monitor disk I/O: `iostat -x 1`
- Check HDD health: `sudo smartctl -a /dev/sdb`
### Backup Disk Full
- Old backups auto-delete after 30 days
- Manually remove old backups if needed:
```bash
# List backups by size
du -h /mnt/media/backups/docker-configs/*
# Remove specific backup
rm /mnt/media/backups/docker-configs/docker-configs-20251001-*.tar.gz
```
### Restore Failed
1. Check backup file integrity:
```bash
tar -tzf /mnt/media/backups/docker-configs/backup-file.tar.gz > /dev/null
```
2. If corrupted, try previous backup
3. Check disk space before restoring:
```bash
df -h ~/docker-data
```
## Backup Storage
**Current Usage:**
- ~94 MB per daily backup
- 30 days retention = ~2.8 GB total
- Stored on 3.6 TB HDD (plenty of space)
**Offsite Backups (Recommended):**
For extra protection, periodically copy backups to external drive:
```bash
# Copy last 7 days to external drive
rsync -av --progress /mnt/media/backups/docker-configs/ /mnt/external-drive/backups/
```
---
**Last Updated:** 2025-11-11
docs/reference/AUTOMATION.md
-343
View File
@@ -1,343 +0,0 @@
# Stack Automation Guide
## Overview
The `update-stack.sh` script enables programmatic stack updates via Portainer's REST API. This allows LLM agents (like Claude) and automation scripts to safely update Portainer stacks without requiring UI access.
## Quick Start
```bash
# Navigate to stacks directory
cd /home/jpmschweitzer/Projects/portainer-core/stacks
# Update a stack (interactive mode - first time)
./update-stack.sh open-webui.yml
# Subsequent updates (uses stored token)
./update-stack.sh open-webui.yml
```
## How It Works
### Authentication Flow
1. **First Run:**
- Prompts for Portainer username/password
- Authenticates with Portainer API
- Generates JWT access token
- Saves token to `.portainer-token` (gitignored)
2. **Subsequent Runs:**
- Reads token from `.portainer-token`
- Uses token for API calls
- No credential prompts needed
### Update Process
1. Reads YAML file from `stacks/` directory
2. Authenticates with Portainer (or uses cached token)
3. Looks up stack by name (filename without .yml)
4. Sends updated stack configuration via API
5. Portainer validates and applies changes
## Usage Modes
### Interactive Mode (Human Operators)
```bash
./update-stack.sh open-webui.yml
```
**First run prompts for:**
- Portainer username
- Portainer password
**Token persists for subsequent runs.**
### Non-Interactive Mode (Automation/LLM Agents)
```bash
export PORTAINER_USERNAME="admin"
export PORTAINER_PASSWORD="your-secure-password"
./update-stack.sh open-webui.yml
```
**Use this mode for:**
- CI/CD pipelines
- LLM agent workflows
- Automated deployment scripts
- Cron jobs
### Environment Variables
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `PORTAINER_URL` | No | `http://localhost:8080` | Portainer instance URL |
| `PORTAINER_USERNAME` | Non-interactive only | - | Admin username |
| `PORTAINER_PASSWORD` | Non-interactive only | - | Admin password |
## Examples
### Update Single Stack
```bash
./update-stack.sh open-webui.yml
```
### Update Multiple Stacks
```bash
for stack in open-webui.yml ollama.yml core-api.yml; do
./update-stack.sh "$stack"
echo "---"
done
```
### LLM Agent Integration
```bash
# Claude Code workflow example
export PORTAINER_USERNAME="admin"
export PORTAINER_PASSWORD="${PORTAINER_ADMIN_PASSWORD}" # from secure env
# Update stack after modifying YAML
./update-stack.sh open-webui.yml
# Check result
echo $? # 0 = success, 1 = failure
```
### Remote Portainer Instance
```bash
export PORTAINER_URL="https://portainer.example.com"
./update-stack.sh my-stack.yml
```
## Security Considerations
### Token Storage
- Token stored in `.portainer-token` (gitignored)
- File permissions: `600` (owner read/write only)
- Token expires based on Portainer settings (default: 8 hours)
- Re-authentication automatic if token expires
### Credentials
**DO NOT:**
- ❌ Commit `.portainer-token` to git
- ❌ Hardcode passwords in scripts
- ❌ Share tokens between users
- ❌ Use root/admin account for automation (create dedicated API user)
**DO:**
- ✅ Use environment variables for non-interactive mode
- ✅ Store credentials in secure password manager
- ✅ Create dedicated Portainer user for automation
- ✅ Rotate passwords regularly
- ✅ Use `.gitignore` to exclude token file
### Best Practices
1. **Create Automation User:**
```
Portainer → Users → Add User
Username: portainer-automation
Role: Environment Administrator (or custom)
```
2. **Use Environment Variables:**
```bash
# In ~/.bashrc or secure environment
export PORTAINER_USERNAME="portainer-automation"
export PORTAINER_PASSWORD="$(pass show portainer/automation)" # from password manager
```
3. **Restrict Permissions:**
- Grant minimum required permissions
- Limit to specific environments/stacks if possible
## Troubleshooting
### Authentication Failed
```
[ERROR] Failed to authenticate. Check credentials and try again.
```
**Solutions:**
- Verify username/password are correct
- Check Portainer is accessible: `curl http://localhost:8080/api/status`
- Ensure user has admin/environment admin role
- Try removing `.portainer-token` and re-authenticating
### Stack Not Found
```
[ERROR] Stack 'my-stack' not found in Portainer
Available stacks:
- open-webui
- ollama
- core-api
```
**Solutions:**
- Verify stack name matches filename (without .yml)
- Check stack exists in Portainer UI
- Stack name is case-sensitive
- Create stack in Portainer first if it doesn't exist
### Connection Refused
```
[ERROR] Failed to connect to Portainer at http://localhost:8080
```
**Solutions:**
- Check Portainer is running: `docker ps | grep portainer`
- Verify port: Portainer default is 8080
- Set `PORTAINER_URL` if using different port/host
- Check firewall rules if accessing remote instance
### Token Expired
```
[ERROR] Invalid authentication token
```
**Solutions:**
- Delete token file: `rm .portainer-token`
- Re-run script to re-authenticate
- Check Portainer token expiration settings
### YAML Validation Error
```
[ERROR] Stack update failed: invalid compose file
```
**Solutions:**
- Validate YAML syntax: `yamllint open-webui.yml`
- Check Docker Compose version compatibility
- Review Portainer logs: `docker logs portainer`
- Test with `docker compose config -f open-webui.yml`
## Integration with LLM Agents
### Claude Code Workflow
This script is designed to integrate seamlessly with Claude Code workflows:
1. **Agent modifies YAML file:**
```python
# Claude uses Edit tool to update open-webui.yml
```
2. **Agent calls update script:**
```bash
cd /home/jpmschweitzer/Projects/portainer-core/stacks
./update-stack.sh open-webui.yml
```
3. **Agent verifies deployment:**
```bash
docker logs open-webui --tail 20
curl http://localhost:82 # Verify service
```
### Setting Up for Claude
Add to user profile or environment:
```bash
# In ~/.bashrc or secure location
export PORTAINER_USERNAME="admin"
export PORTAINER_PASSWORD="your-secure-password"
# Or use password manager
export PORTAINER_PASSWORD="$(pass show portainer/admin)"
```
Then Claude can directly call:
```bash
./update-stack.sh <stack-file.yml>
```
## Advanced Usage
### Custom Portainer URL
```bash
# Connect to remote Portainer
export PORTAINER_URL="https://portainer.mydomain.com"
./update-stack.sh open-webui.yml
```
### Token Management
```bash
# View current token (for debugging)
cat .portainer-token | base64 -d | jq
# Force re-authentication
rm .portainer-token
./update-stack.sh open-webui.yml
# Use specific token
echo "your-jwt-token-here" > .portainer-token
chmod 600 .portainer-token
```
### Dry Run (Check Only)
```bash
# Validate YAML before updating
docker compose -f open-webui.yml config
# Check current stack status
curl -s http://localhost:8080/api/stacks \
-H "Authorization: Bearer $(cat .portainer-token)" | jq
```
## API Reference
The script uses these Portainer API endpoints:
- `POST /api/auth` - Authenticate and get token
- `GET /api/endpoints` - List Docker endpoints
- `GET /api/stacks` - List all stacks
- `PUT /api/stacks/{id}` - Update specific stack
For full API documentation: https://docs.portainer.io/api/docs
## Maintenance
### Regular Tasks
- **Monthly:** Rotate automation user password
- **Quarterly:** Review and audit API access logs
- **After incidents:** Revoke and regenerate tokens
### Token Rotation
```bash
# Revoke old token (Portainer UI)
Portainer → Users → [user] → Access Tokens → Revoke All
# Re-authenticate
rm .portainer-token
./update-stack.sh open-webui.yml
```
## Support
For issues or questions:
1. Check Portainer logs: `docker logs portainer`
2. Review this guide's Troubleshooting section
3. Check Portainer API docs: https://docs.portainer.io/api/docs
4. Open issue in project repository
---
*Last updated: 2025-11-14*
docs/reference/SYSTEM.md
-266
View File
@@ -1,266 +0,0 @@
# SYSTEM.md
**System documentation for LLM coding agents** - This file describes the computer system where this project resides, including hardware, OS, installed software, and environment details.
> Last updated: 2025-11-11
## System Overview
- **Hostname**: tower-of-joy
- **User**: jpmschweitzer
- **Home Directory**: /home/jpmschweitzer
- **Project Location**: /home/jpmschweitzer/Projects/portainer-core
## Operating System
### Distribution
- **OS**: Zorin OS 16.3
- **Based on**: Ubuntu 20.04 (Focal Fossa)
- **Kernel**: Linux 5.4.0-216-generic
- **Architecture**: x86_64 (64-bit)
### Desktop Environment
- **Display Server**: X11 (GDM)
- **Desktop**: GNOME Shell (Zorin session mode)
- **Session Manager**: gnome-session
### Locale & Timezone
- **Language**: en_US.UTF-8
- **Numeric/Time Format**: nl_NL.UTF-8
- **Timezone**: Europe/Amsterdam (CET, +0100)
## Hardware Specifications
### CPU
- **Model**: Intel Core i7-6700 @ 3.40GHz (6th Gen Skylake)
- **Cores**: 4 physical cores, 8 threads (2 threads per core)
- **Architecture**: x86_64
- **Frequency**: 800 MHz - 4000 MHz (currently ~3666 MHz)
- **Cache**:
- L1d: 128 KiB
- L1i: 128 KiB
- L2: 1 MiB
- L3: 8 MiB
- **Virtualization**: VT-x supported
- **Notable Flags**: AVX, AVX2, AES-NI, SSE4.1, SSE4.2, FMA
### Memory
- **Total RAM**: 16 GiB
- **Available**: ~9.6 GiB (typical)
- **Swap**: 2.0 GiB
### Storage
**System has 2 disks with total capacity of 4.2TB:**
#### Disk 1: System SSD (/dev/sda)
- **Model**: Crucial CT525MX300SSD1 (525GB SSD)
- **Partition**: /dev/sda1
- **Filesystem**: ext4
- **Total Size**: 489 GB
- **Used**: 92 GB (21%)
- **Available**: 365 GB
- **Mount Point**: `/` (root)
- **Purpose**: Operating system, Docker containers, application data
#### Disk 2: Media HDD (/dev/sdb)
- **Model**: Seagate IronWolf NE ST4000NE001 (4TB NAS-grade HDD)
- **Total Size**: 3.7 TB
- **Filesystem**: ext4
- **Label**: "media"
- **UUID**: f4300e91-3f51-45a0-b038-03335c5bd792
- **Mount Status**: ⚠️ **Currently unmounted** (not in /etc/fstab)
- **Purpose**: Media storage for Jellyfin, Nextcloud data, backups
- **Drive Type**: NAS-optimized (24/7 operation, multi-user workloads)
**Total Storage Capacity**: 4.2 TB
### Graphics
- **GPU**: NVIDIA GeForce RTX 2080 Ti (TU102, Rev. A)
- **VRAM**: 11 GB (11018 MiB)
- **Driver**: NVIDIA 470.256.02
- **CUDA Version**: 11.4
- **Bus**: PCIe 0a:00.0
- **Current Usage**: ~390 MiB VRAM (mostly X11/GNOME)
- **Power**: 260W TDP
**Note**: NVCC (CUDA compiler) is not currently in PATH, but CUDA drivers are installed.
## Development Tools & Languages
### Programming Languages
#### Python
- **Version**: 3.8.10 (system default)
- **pip**: 25.3 (Python 3.10 in user site-packages)
- **Location**: /usr/bin/python3
- **Python 2.x**: Not installed
- **Virtual Environments**:
- virtualenv: Not installed
- Conda: Not installed
- venv module: Available (built-in)
#### Node.js & JavaScript
- **Node.js**: v24.11.0
- **npm**: 11.6.1
- **Version Manager**: NVM installed at /home/jpmschweitzer/.nvm
#### Java
- **Version**: Java 21.0.4 LTS (Oracle JDK)
- **Runtime**: Java(TM) SE Runtime Environment (build 21.0.4+8-LTS-274)
- **VM**: Java HotSpot 64-Bit Server VM
#### C/C++
- **GCC**: 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)
- **Make**: GNU Make 4.2.1
- **CMake**: Not installed
#### Other Languages
- **Go**: Not installed
- **Rust**: Not installed
### Version Control
- **Git**: 2.25.1
### Containerization & Virtualization
- **Docker**: 28.1.1, build 4eba377
### Editors & IDEs
- **Vim**: 8.1 (2018 May 18)
- **VS Code**: Not installed
### Command Line Tools
- **Shell**: Bash 5.0.17
- **curl**: 7.68.0
- **wget**: 1.20.3
- **SSH**: OpenSSH 8.2p1 Ubuntu-4ubuntu0.13
## GPU & CUDA Information
### NVIDIA GPU Details
The system has an NVIDIA RTX 2080 Ti with CUDA support, suitable for:
- Machine learning and deep learning workloads
- CUDA-accelerated computing
- GPU rendering and compute tasks
- Parallel processing
### CUDA Configuration
- **Driver Version**: 470.256.02
- **CUDA Toolkit Version**: 11.4 (driver supports)
- **Compute Capability**: 7.5 (Turing architecture)
- **NVCC**: Not in PATH (may need manual setup)
### GPU Usage Considerations
When working with GPU-accelerated code:
- Ensure CUDA toolkit is properly installed if needed
- Use appropriate CUDA version compatibility (11.4 or compatible)
- PyTorch/TensorFlow should use CUDA 11.x compatible builds
- Monitor VRAM usage (11 GB total, ~10.6 GB available for compute)
## System Capabilities & Recommendations
### Suitable For
- **Web Development**: Node.js, npm available
- **Python Development**: Python 3.8 with pip
- **Java Development**: Java 21 LTS
- **Machine Learning**: CUDA-capable GPU with 11GB VRAM
- **Containerized Development**: Docker available
- **Compiled Languages**: GCC toolchain available
- **Media Server**: 3.7TB NAS-grade storage for Jellyfin/Plex
- **NAS/File Server**: Seagate IronWolf drive optimized for 24/7 operation
- **Cloud Storage**: Ample space for Nextcloud deployments
- **Home Server**: Suitable for comprehensive home lab setup
### Limitations
- No Rust toolchain (needs installation)
- No Go compiler (needs installation)
- CMake not installed (needed for some C/C++ projects)
- VS Code not installed (Vim available as alternative)
- CUDA compiler not in PATH
### Environment Notes
- NVM is available for Node.js version management
- Python 3.8 is the system default (older, consider pyenv for newer versions)
- pip is installed in user site-packages (Python 3.10 version)
- Docker is available for containerized workflows
## Package Management
### System Package Manager
- **APT**: Available (Ubuntu/Debian package manager)
- Use `sudo apt install <package>` for system packages
### Language-Specific Package Managers
- **Python**: pip3 (25.3)
- **Node.js**: npm (11.6.1), managed via NVM
- **Java**: Maven/Gradle likely needed (not verified)
## Network Information
- SSH client available (OpenSSH 8.2p1)
- Standard network tools available (curl, wget)
## Usage Notes for LLM Agents
### Before Installing New Software
1. Check if the tool is already installed using `which <command>`
2. Check available disk space:
- System SSD: 365 GB available (for OS and containers)
- Media HDD: 3.7 TB available (currently unmounted - needs mounting)
3. Use appropriate package manager (apt, pip, npm, etc.)
4. Consider using Docker for isolated environments
5. **Mount the 4TB media drive** before deploying data-intensive services:
- Recommended mount point: `/mnt/media` or `/media/storage`
- Add to `/etc/fstab` for automatic mounting on boot
- UUID: `f4300e91-3f51-45a0-b038-03335c5bd792`
### GPU Development
1. Verify CUDA toolkit path if developing GPU code
2. Check GPU memory availability with `nvidia-smi`
3. Use CUDA 11.x compatible libraries
4. Monitor GPU utilization to avoid OOM errors
### Python Development
1. System Python is 3.8.10 (older version)
2. Consider using venv for project isolation
3. pip is available but points to Python 3.10 libs in user space
4. May need to install python3-venv: `sudo apt install python3-venv`
### Node.js Development
1. NVM is installed for version management
2. Current Node.js is v24.11.0 (latest as of 2024)
3. npm 11.6.1 is available
### Docker Usage
1. Docker 28.1.1 is installed
2. Useful for consistent development environments
3. Can isolate dependencies and avoid system conflicts
### Storage Management (Dual-Disk Setup)
1. **System SSD (/dev/sda)**: Use for:
- Operating system
- Docker images and container configs
- Application databases (small, performance-critical)
- Cache directories
2. **Media HDD (/dev/sdb)**: Use for:
- Jellyfin/Plex media libraries
- Nextcloud user data
- Backups and archives
- Large file storage
- Any data-intensive workloads
3. **Best Practices**:
- Keep Docker container configs on SSD for performance
- Store media files on HDD (they're sequential access, HDD is fine)
- Use bind mounts to map HDD storage into containers
- Example: `-v /mnt/media/jellyfin:/media:ro` in Docker
4. **Before First Use**:
- Mount the media drive (see step 5 in "Before Installing New Software")
- Verify mount with `df -h /mnt/media`
- Set appropriate permissions: `sudo chown -R $USER:$USER /mnt/media`
---
*Generated automatically on 2025-11-11, updated with storage configuration*
*For project-specific guidelines, see [AGENTS.md](./AGENTS.md)*
docs/reference/scripts.md
-200
View File
@@ -1,200 +0,0 @@
# Maintenance Scripts Reference
Shell scripts for common maintenance tasks located in `/scripts/`.
## Available Scripts
| Script | Description | Usage |
|--------|-------------|-------|
| `gpu-check.sh` | Verify GPU passthrough in containers | `./scripts/gpu-check.sh` |
| `health-check.sh` | Check all services and report status | `./scripts/health-check.sh` |
| `setup-kuma-monitors.sh` | Manual guide for configuring Uptime Kuma monitors | `./scripts/setup-kuma-monitors.sh` |
| `setup-kuma-monitors.py` | **Automated** Uptime Kuma monitor setup via API | `source .venv/bin/activate && python3 scripts/setup-kuma-monitors.py` |
| `backup-configs.sh` | Backup all Docker configs | `./scripts/backup-configs.sh` |
| `disk-usage.sh` | Report disk usage for SSD and HDD | `./scripts/disk-usage.sh` |
| `update-stacks.sh` | Pull latest images and update containers | `./scripts/update-stacks.sh <stack-name>` |
| `cleanup.sh` | Clean up unused Docker resources | `./scripts/cleanup.sh` |
## Making Scripts Executable
```bash
# Make all scripts executable
chmod +x scripts/*.sh
# Or individually
chmod +x scripts/health-check.sh
```
## Scheduling with Cron
Add to crontab for automated maintenance:
```bash
# Edit crontab
crontab -e
# Examples:
# Daily health check at 8 AM
0 8 * * * /home/jpmschweitzer/Projects/portainer-core/scripts/health-check.sh >> /var/log/portainer-core-health.log 2>&1
# Weekly cleanup on Sunday at 3 AM
0 3 * * 0 /home/jpmschweitzer/Projects/portainer-core/scripts/cleanup.sh
# Daily backup at 2 AM
0 2 * * * /home/jpmschweitzer/Projects/portainer-core/scripts/backup-configs.sh
```
## Script Details
### GPU Check (`gpu-check.sh`)
Verifies GPU passthrough is working in GPU-enabled containers (Ollama, Jellyfin).
**Usage:**
```bash
./scripts/gpu-check.sh
```
**Output:**
- Lists all running containers with GPU access
- Runs `nvidia-smi` inside each container
- Reports any containers that fail GPU detection
### Health Check (`health-check.sh`)
Checks status of all deployed services and generates a health report.
**Usage:**
```bash
./scripts/health-check.sh
```
**Checks:**
- Container running status
- Container health status (if health check defined)
- Port accessibility
- Basic connectivity tests
### Uptime Kuma Monitor Setup
Two versions available:
**Manual Script (`setup-kuma-monitors.sh`):**
- Interactive guide for adding monitors
- Shows recommended settings for each service
- Good for understanding monitor configuration
**Automated Script (`setup-kuma-monitors.py`):**
- Python script using Uptime Kuma API
- Automatically creates monitors for all services
- Requires Uptime Kuma API key
**Usage:**
```bash
# Automated setup
source .venv/bin/activate
python3 scripts/setup-kuma-monitors.py
```
### Backup Configs (`backup-configs.sh`)
Backs up Docker container configurations and important data.
**Usage:**
```bash
./scripts/backup-configs.sh
```
**What it backs up:**
- Docker Compose files from `/stacks/`
- Container configs from `/home/jpmschweitzer/docker-data/`
- Project documentation
- Excludes large media files (those are backed up separately)
**Backup location:**
- `/mnt/media/backups/portainer-core/`
See [Backup Procedures](../guides/backup-procedures.md) for comprehensive backup strategy.
### Disk Usage (`disk-usage.sh`)
Reports disk usage breakdown for SSD and HDD storage.
**Usage:**
```bash
./scripts/disk-usage.sh
```
**Output:**
- Total SSD usage (`/home/jpmschweitzer/docker-data/`)
- Total HDD usage (`/mnt/media/`)
- Per-service breakdown
- Available space warnings
### Update Stacks (`update-stacks.sh`)
Pulls latest images and updates a specific stack.
**Usage:**
```bash
./scripts/update-stacks.sh <stack-name>
# Examples:
./scripts/update-stacks.sh jellyfin
./scripts/update-stacks.sh core-api
```
**What it does:**
1. Pulls latest images for the stack
2. Stops containers gracefully
3. Recreates containers with new images
4. Removes old images
5. Verifies containers started successfully
**Note:** Watchtower handles this automatically for most services. Use this script for manual updates or services excluded from Watchtower.
### Cleanup (`cleanup.sh`)
Cleans up unused Docker resources to free disk space.
**Usage:**
```bash
./scripts/cleanup.sh
```
**What it removes:**
- Stopped containers
- Unused images
- Dangling build cache
- Unused volumes (with confirmation prompt)
- Unused networks
**Warning:** Always review what will be removed before confirming volume deletion.
## Script Guidelines
All scripts follow these conventions:
- Include error handling and exit codes
- Use absolute paths for reliability
- Log output for debugging
- Exit with status codes (0 = success, non-zero = failure)
- Include help text with `-h` or `--help` flags
- Non-destructive by default (ask before deleting)
## Creating New Scripts
When adding new maintenance scripts:
1. Place in `/scripts/` directory
2. Use `.sh` extension for shell scripts
3. Make executable: `chmod +x scripts/your-script.sh`
4. Add to this documentation
5. Include help text and error handling
6. Test thoroughly before scheduling with cron
## Related Documentation
- [Backup Procedures](../guides/backup-procedures.md) - Comprehensive backup strategy
- [Stacks Reference](stacks.md) - Stack deployment and management
- [Automation Reference](AUTOMATION.md) - Portainer REST API automation
docs/reference/stacks.md
-185
View File
@@ -1,185 +0,0 @@
# Docker Compose Stacks Reference
Complete reference for all Docker Compose stacks in the portainer-core infrastructure.
## Deployment
See the [core-api OpenAPI documentation](http://localhost:8083/docs) for infrastructure management REST endpoints.
All stacks are located in the `/stacks/` directory and version-controlled.
## Stack Inventory
### Phase 1: Foundation
| Stack | File | Ports | GPU | Description |
|-------|------|-------|-----|-------------|
| **Portainer** | `portainer.yml` | 8080, 8443 | No | Container management UI |
| **Nginx Proxy Manager** | `nginx-proxy-manager.yml` | 8000, 80, 443 | No | Reverse proxy and unified web interface |
| **Ollama** | `ollama.yml` | 11434 | **Yes** | ML model serving with GPU acceleration |
### Phase 2: Networking
| Stack | File | Ports | GPU | Description |
|-------|------|-------|-----|-------------|
| **Headscale** | `headscale.yml` | 8085, 9090 | No | Self-hosted Tailscale control server |
### Phase 3: Monitoring
| Stack | File | Ports | GPU | Description |
|-------|------|-------|-----|-------------|
| **Uptime Kuma** | `uptime-kuma.yml` | 3001 | No | Service availability monitoring |
| **Netdata** | `netdata.yml` | 19999 | No | Real-time system performance monitoring |
| **Heimdall** | `heimdall.yml` | 8888, 8889 | No | Application dashboard |
### Phase 4: Optimization
| Stack | File | Ports | GPU | Description |
|-------|------|-------|-----|-------------|
| **Watchtower** | `watchtower.yml` | - | No | Automatic container updates |
| **Duplicati** | `duplicati.yml` | 8200 | No | Backup solution |
### Applications
| Stack | File | Ports | GPU | Description |
|-------|------|-------|-----|-------------|
| **Jellyfin** | `jellyfin.yml` | 8096, 8920, 7359, 1900 | **Yes** | Media server with GPU transcoding |
| **Nextcloud** | `nextcloud.yml` | 8082 | No | Cloud storage (uses shared PostgreSQL and Redis) |
| **Gitea** | `gitea.yml` | 3002, 2222 | No | Git repository hosting (includes PostgreSQL) |
| **Samba** | `samba.yml` | 139, 445 | No | Network file sharing |
| **Open WebUI** | `open-webui.yml` | 8081 | No | AI chat interface with Ollama integration |
| **Core API** | `core-api.yml` | 8083 | No | Infrastructure management and AI orchestration |
| **Qdrant** | `qdrant.yml` | 6333, 6334 | No | Vector database for embeddings |
| **Organizr** | `organizr.yml` | 8084 | No | Unified dashboard |
### Shared Infrastructure
| Stack | File | Ports | GPU | Description |
|-------|------|-------|-----|-------------|
| **PostgreSQL Shared** | `postgres-shared.yml` | 5432 | No | Shared database for Nextcloud |
| **Redis Shared** | `redis-shared.yml` | 6379 | No | Shared cache for Nextcloud |
## Port Allocation
### Infrastructure Services (8000-8099)
- 8000: Nginx Proxy Manager (unified web interface)
- 8080: Portainer
- 8081: Open WebUI
- 8082: Nextcloud
- 8083: Core API
- 8084: Organizr
- 8085: Headscale
- 8096: Jellyfin
### Git & Development Services
- 2222: Gitea SSH
- 3002: Gitea HTTP
### Monitoring Services (3000-3999, 19000-19999)
- 3001: Uptime Kuma
- 8200: Duplicati
- 8888: Heimdall
- 19999: Netdata
### ML/API Services (11000+)
- 11434: Ollama
- 6333: Qdrant HTTP
- 6334: Qdrant gRPC
### Database Services
- 5432: PostgreSQL (shared)
- 6379: Redis (shared)
### Network Services
- 80: HTTP (NPM reverse proxy)
- 443: HTTPS (NPM reverse proxy)
- 139, 445: Samba/SMB
- 9090: Headscale metrics
## Storage Convention
All stacks follow the dual-disk strategy:
**SSD (Performance):**
- Configs: `/home/jpmschweitzer/docker-data/<service>/config`
- Cache: `/home/jpmschweitzer/docker-data/<service>/cache`
- Databases: `/home/jpmschweitzer/docker-data/<service>/db`
**HDD (Capacity):**
- User content: `/mnt/media/<service>/data`
- Media files: `/mnt/media/<service>/media`
- Backups: `/mnt/media/backups/<service>`
See [Shared Infrastructure Architecture](../architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md) for database and cache sharing details.
## GPU Services
Stacks requiring GPU access (marked with **Yes** above):
- `ollama.yml` - ML model inference
- `jellyfin.yml` - Hardware transcoding
**Prerequisites:**
- NVIDIA Container Toolkit installed
- GPU verified: `docker run --rm --gpus all nvidia/cuda:11.4.0-base-ubuntu20.04 nvidia-smi`
See [GPU Docker Configuration](../guides/gpu-docker-config.md) for setup details.
## Deployment Checklist
### Before Deploying
1. **Review environment variables** - Change default passwords!
2. **Create directories** - Ensure volume paths exist
3. **Check ports** - Verify no conflicts with existing services
4. **GPU services** - Confirm NVIDIA toolkit installed
5. **Update STATUS.md** - Plan the deployment
### After Deploying
1. **Test service** - Access web UI or API endpoint
2. **Check logs** - `docker logs <container-name>`
3. **Verify GPU** - `docker exec <container> nvidia-smi` (if applicable)
4. **Update documentation** - Add to STATUS.md and CHANGELOG.md
5. **Configure backup** - Add to Duplicati backup job
6. **Add monitoring** - Configure Uptime Kuma checks
## Maintenance
### Update a Stack
```bash
# Pull latest images
docker compose -f stacks/<stack-name>.yml pull
# Recreate containers with new images
docker compose -f stacks/<stack-name>.yml up -d
# Or let Watchtower handle it automatically
```
### Backup Stack Configuration
Stacks are version-controlled in the `/stacks/` directory. Backup container data separately using the backup procedures.
See [Backup Procedures](../guides/backup-procedures.md) for details.
### Troubleshooting
- Container won't start: `docker logs <container-name>`
- Port conflicts: `sudo netstat -tulpn | grep <port>`
- Permission issues: Check volume path ownership
- GPU not detected: Verify NVIDIA toolkit and restart Docker
## Automation
The project includes automation scripts for stack management:
- `update-stack.sh` - Pull and update specific stack
- See [Automation Reference](AUTOMATION.md) for Portainer REST API usage
## Related Documentation
- [Container Reference](CONTAINERS.md) - Complete container profiles
- [System Specifications](SYSTEM.md) - Hardware and software specs
- [Shared Infrastructure Architecture](../architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md) - Database/cache sharing
- [Maintenance Scripts](scripts.md) - Automated maintenance tasks
docs/sessions/2025-11-20-authentik-deployment.md
-409
View File
@@ -1,409 +0,0 @@
# Authentik SSO Deployment Session
**Date:** 2025-11-20
**Duration:** ~4 hours
**Status:** Milestone 2/5 Complete (Google OAuth Working)
**Version:** 0.8.0-authentik-sso
## Session Overview
Successfully deployed Authentik identity provider with Google OAuth integration and optimized memory usage. Forward authentication configuration blocked on embedded outpost initialization issue.
---
## Accomplishments
### ✅ Milestone 1: Authentik Deployment (COMPLETE)
**Infrastructure Setup:**
- Deployed Authentik server and worker containers (version 2024.8.4)
- Configured shared PostgreSQL: `authentik` database with `authentik_user`
- Configured shared Redis: Database 0
- Network: Connected to `docker-dataplane`
**Configuration Highlights:**
```yaml
Memory Limits:
- Server: 512M limit, 256M reservation
- Worker: 384M limit, 128M reservation
- Total: 563MB actual usage (vs 3-5GB previous attempt = 80-90% reduction!)
Ports:
- 9000: Web UI
- 9444: Embedded outpost (mapped from container 9443)
Environment:
- AUTHENTIK_HOST: https://auth.schweitz.net
- AUTHENTIK_COOKIE_DOMAIN: .schweitz.net
- PostgreSQL: postgres-shared:5432/authentik
- Redis: redis-shared:6379/0
```
**Issues Resolved:**
1. **Health check failure** - Container didn't have wget/curl
- Solution: Used Python's urllib.request for health checks
2. **Database user didn't exist** - authentik_user not created by init script
- Solution: Manually created user with proper grants
3. **Port conflict** - 9443 already in use
- Solution: Mapped to 9444 on host
4. **NPM proxy missing** - auth.schweitz.net not visible in UI
- Solution: Entry was marked as deleted (is_deleted=1), recreated via UI
**NPM Configuration:**
- Created proxy host for auth.schweitz.net
- Forward to: http://localhost:9000
- SSL: Let's Encrypt (enforced, HSTS enabled)
- **Critical:** NO forward auth on auth.schweitz.net (prevents redirect loops)
### ✅ Milestone 2: Google OAuth Integration (COMPLETE)
**Google Cloud Console Setup:**
- Created OAuth credentials:
- Client ID: `59195574918-813nsfslhjduqto8nc4a3ejg2lj133il.apps.googleusercontent.com`
- Client Secret: `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`
- Authorized redirect URI: `https://auth.schweitz.net/source/oauth/callback/google/`
**Authentik Configuration (via API):**
```python
# Created Google OAuth source
Source: "Google"
Slug: "google"
Provider: "google"
Consumer Key: [Google Client ID]
Consumer Secret: [Google Client Secret]
Enrollment Flow: default-source-enrollment
Authentication Flow: default-source-authentication
```
**Login Flow Configuration:**
- Updated `default-authentication-identification` stage
- Enabled "Show sources' labels"
- Added Google source to sources list
- Result: Google login button now appears on login page
**Testing Results:**
- ✅ Google login button visible on auth.schweitz.net
- ✅ OAuth redirect to Google works
- ✅ User created successfully: `jpmschweitzer@gmail.com`
- ✅ User type: `external` (correct for OAuth users)
- ⚠️ External users blocked from admin interface (expected behavior)
- ✅ Admin access via `akadmin` recovery key
**Enrollment Flow Issue & Resolution:**
- Initial error: "Flow does not apply to current user"
- Root cause: Browser session had conflicting flow plan cached
- Solution: Cleared cookies, used incognito window
- Policy check: `default-source-enrollment-if-sso` working correctly
### 🚧 Milestone 3: Forward Auth for Organizr (BLOCKED)
**Progress:**
- ✅ Created Proxy Provider "Organizr Proxy" via API
- Mode: `forward_single`
- External host: `https://home.schweitz.net`
- Authorization flow: `default-provider-authorization-implicit-consent`
- ✅ Created Application "Organizr" via API
- Slug: `organizr`
- Provider: Organizr Proxy
- Launch URL: `https://home.schweitz.net`
- ✅ Assigned provider to embedded outpost
- ✅ Embedded outpost responding on port 9444
- Ping endpoint works: `https://localhost:9444/outpost.goauthentik.io/ping`
**Current Blocker:**
```
Issue: Auth endpoint returns 404
Endpoint: https://localhost:9444/outpost.goauthentik.io/auth/nginx
Status: 404 Not Found
Expected: 200 OK or 401/302 for unauthenticated requests
NPM Error Logs:
auth request unexpected status: 404 while sending to client
```
**Analysis:**
- Outpost is running and healthy
- Ping endpoint responds correctly
- Auth endpoint not being exposed by outpost
- Possible causes:
1. Provider mode issue (`forward_single` vs `forward_domain`)
2. Outpost not loading provider configuration
3. Auth endpoint path incorrect for Authentik 2024.8.4
4. Embedded outpost initialization incomplete
**Forward Auth Config Attempted:**
```nginx
# NPM advanced config for home.schweitz.net
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;
location /outpost.goauthentik.io {
proxy_pass https://localhost:9444/outpost.goauthentik.io;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
# ... (additional headers)
}
location @goauthentik_proxy_signin {
internal;
return 302 /outpost.goauthentik.io/start?rd=$request_uri;
}
```
**Config Reverted:**
- Restored original NPM config for home.schweitz.net
- Organizr accessible without SSO (for now)
- Backup saved: `/data/nginx/proxy_host/2.conf.backup`
---
## Technical Details
### API Usage
Successfully used Authentik's REST API for automation:
```bash
# Created temporary API token
Token: dbc4eda544fd141a015b1ad1ec42955a4f6666fd22456a88c6f6402afa3107d1
Duration: 1 hour
User: akadmin
# API Endpoints Used:
POST /api/v3/providers/proxy/ # Create provider
POST /api/v3/core/applications/ # Create application
PATCH /api/v3/outposts/instances/{id}/ # Assign provider to outpost
GET /api/v3/flows/instances/ # List flows
```
### Database Operations
```sql
-- Created authentik database and user
CREATE DATABASE authentik;
CREATE USER authentik_user WITH PASSWORD 'F//j0ktck7cX06Vfgh0YXceONOtlSsHvadqROICeDx8=';
GRANT ALL PRIVILEGES ON DATABASE authentik TO authentik_user;
GRANT ALL ON SCHEMA public TO authentik_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO authentik_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO authentik_user;
-- Verified user creation
SELECT id, username, email, is_active, type
FROM authentik_core_user
WHERE email = 'jpmschweitzer@gmail.com';
-- Result: id=5, type=external, is_active=t
-- Checked OAuth source
SELECT slug, name, enabled, provider_type
FROM authentik_core_source s
LEFT JOIN authentik_sources_oauth_oauthsource o
ON s.policybindingmodel_ptr_id = o.source_ptr_id;
-- Result: slug=google, enabled=t, provider_type=google
```
### Memory Optimization Success
**Previous Failed Deployment:**
- Memory usage: 3-5GB
- Separate PostgreSQL instance: ~1GB
- Separate Redis instance: ~100MB
- No resource limits
**Current Deployment:**
```bash
$ docker stats authentik-server authentik-worker --no-stream
NAME CPU % MEM USAGE / LIMIT MEM %
authentik-server 0.52% 291.1MiB / 512MiB 56.85%
authentik-worker 2.87% 271.9MiB / 384MiB 70.80%
Total: ~563MB
Savings: 82-88% reduction
Strategy:
- Shared PostgreSQL (no dedicated instance)
- Shared Redis (no dedicated instance)
- Resource limits enforced
- Single worker with 2 threads
- Disabled: avatars, error reporting, footer links
- Log level: warning
```
### Files Modified
1. **[stacks/authentik.yml](../../stacks/authentik.yml)** - Created
- Authentik server and worker configuration
- Shared infrastructure connections
- Resource limits and health checks
- Port mappings: 9000, 9444
2. **NPM Database** - Modified
- Created proxy host for auth.schweitz.net
- Attempted forward auth config (reverted)
3. **PostgreSQL** - Modified
- Created authentik database
- Created authentik_user with grants
4. **[STATUS.md](../../STATUS.md)** - Updated
- Version: 0.8.0-authentik-sso
- Active work: Security & SSO Implementation
- Added Milestone 1 & 2 accomplishments
- Documented Milestone 3 blocker
---
## Known Issues
### 1. Embedded Outpost Auth Endpoint Not Working
**Symptom:**
```
curl -k https://localhost:9444/outpost.goauthentik.io/auth/nginx
HTTP/1.1 404 Not Found
```
**Impact:**
- Cannot configure forward authentication for applications
- NPM forward auth results in 500 errors
- Applications remain unprotected
**Possible Solutions:**
1. **Change provider mode:**
```python
# Update via Authentik UI: Applications → Providers → Organizr Proxy
mode: "forward_domain" # instead of "forward_single"
cookie_domain: "schweitz.net"
```
2. **Deploy standalone outpost:**
```yaml
# Add to authentik.yml or separate stack
authentik-proxy:
image: ghcr.io/goauthentik/proxy:2024.8.4
environment:
AUTHENTIK_HOST: https://auth.schweitz.net
AUTHENTIK_TOKEN: <outpost-token>
ports:
- "9443:9443"
```
3. **Wait for full initialization:**
- Monitor logs: `docker logs -f authentik-server`
- Check outpost status in Authentik UI: System → Outposts
- Verify provider assignment
4. **Investigate version compatibility:**
- Authentik 2024.8.4 embedded outpost behavior
- Check if auth endpoint requires specific configuration
- Review Authentik documentation for forward auth setup
### 2. NPM Configuration Persistence
**Issue:**
- Database updates don't trigger nginx config regeneration
- Manual nginx file editing required
- Changes lost on NPM restart/update
**Workaround:**
- Update via NPM UI instead of database direct modification
- Keep backup of custom nginx configs
- Document config in code/scripts for reproducibility
---
## Next Steps
### Immediate (Milestone 3 Completion)
1. **Investigate Outpost Configuration:**
- Check Authentik UI: System → Outposts → authentik Embedded Outpost
- Verify provider is assigned and status is healthy
- Review outpost logs for errors
2. **Try Provider Mode Change:**
- Update Organizr Proxy provider to `forward_domain` mode
- Add `cookie_domain: schweitz.net`
- Restart Authentik containers
- Test auth endpoint again
3. **Alternative: Deploy Standalone Outpost:**
- Create outpost stack configuration
- Generate outpost token in Authentik UI
- Deploy container and test auth endpoint
4. **Test Forward Auth:**
- Once auth endpoint works, apply NPM config
- Test redirect to Authentik login
- Verify SSO session persistence
- Check for redirect loops
### Future Milestones (from security-implementation-plan.md)
- **M4:** Protect Core API with OIDC
- **M5:** Protect remaining services (9 services)
- Jellyfin, Nextcloud, Gitea, Portainer, NPM, Uptime Kuma, Open WebUI, Netdata, Headscale
- **M6:** Documentation and rollback procedures
---
## Lessons Learned
### What Went Well
1. **Shared Infrastructure Approach:**
- Massive memory savings (80-90% reduction)
- Easier management (single PostgreSQL/Redis)
- Successful from day 1
2. **API-Driven Configuration:**
- Faster than UI clicks
- Reproducible and documentable
- Can be scripted for future deployments
3. **Incremental Testing:**
- Validated each component before moving forward
- Caught issues early (health checks, database permissions)
- Easy to rollback when issues encountered
4. **Documentation During Implementation:**
- Captured decisions and solutions in real-time
- Easier to resume work later
- Helpful for troubleshooting
### What Could Be Improved
1. **Version Research:**
- Should have checked Authentik 2024.8.4 embedded outpost capabilities first
- Version 2024.10+ has redirect loop issues (documented in security plan)
- Tradeoff: stability vs features
2. **NPM Configuration Method:**
- Direct database edits don't trigger config regeneration
- Should have used NPM UI from start
- Need better automation for NPM config management
3. **Testing Approach:**
- Should have tested outpost endpoints before configuring NPM
- Could have saved time on troubleshooting
- Need outpost validation checklist
4. **Initialization Timing:**
- Didn't account for embedded outpost startup delay
- Should wait for full health before testing endpoints
- Need patience with complex distributed systems
---
## References
- [Security Implementation Plan](../plans/active/security-implementation-plan.md)
- [Shared Infrastructure Architecture](../architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md)
- [Authentik Documentation](https://goauthentik.io/docs/)
- [NPM Backup](../../backups/npm-database-m0-20251120-152926.sqlite)
- [Authentik Stack](../../stacks/authentik.yml)
---
**Session End Status:**
- ✅ Authentik deployed and accessible
- ✅ Google OAuth fully functional
- ⚠️ Forward auth blocked on outpost initialization
- 🔄 Investigation continuing in next session
docs/sessions/2025-11-21-authentik-troubleshooting.md
-728
View File
@@ -1,728 +0,0 @@
# Authentik Embedded Outpost Troubleshooting Session
**Date:** 2025-11-21
**Session:** Day 3 of Authentik Implementation
**Status:** 🔄 IN PROGRESS - Investigating embedded outpost 404 issue
---
## Session Context
**Previous Session:** [2025-11-20 Authentik Deployment](2025-11-20-authentik-deployment.md)
**Current State:**
- ✅ Authentik deployed (Milestone 1 complete)
- ✅ Google OAuth working (Milestone 2 complete)
- ❌ Forward auth blocked (Milestone 3 blocked on embedded outpost 404)
**Blocker:**
```
Endpoint: http://192.168.86.149:9000/outpost.goauthentik.io/auth/nginx
Status: 404 Not Found
Expected: 401 Unauthorized (for unauthenticated requests)
```
---
## Root Cause Analysis
### 🔍 Research Findings
Conducted comprehensive research of Authentik documentation, GitHub issues, and community implementations. Key findings:
#### 1. **Embedded Outpost Architecture (CRITICAL MISUNDERSTANDING)**
**Previous Understanding (INCORRECT):**
- Embedded outpost runs on separate port 9443/9444
- Port 9000 = Web UI only
- Port 9443 = Outpost endpoints only
**Actual Architecture (CORRECT):**
- Embedded outpost **shares port 9000** with the web UI
- Port 9443 is for **optional TLS termination**, not a separate service
- Outpost uses **path-based routing**: `/outpost.goauthentik.io/*` on port 9000
- The embedded outpost is part of the server process, not a separate container
**Source:**
- Official Authentik docs: "The embedded outpost runs within the server container"
- GitHub issues confirm embedded outpost serves on port 9000
#### 2. **Common Causes of /auth/nginx 404 Error**
From research and GitHub issues:
1. **Missing `/outpost.goauthentik.io` location block in nginx** (most common)
- NPM must proxy this path to Authentik
- Without it, auth_request fails with 404
2. **Provider not assigned to outpost**
- Proxy provider created but not linked to embedded outpost
- Outpost doesn't load provider configuration
- Auth endpoint not exposed
3. **Embedded outpost not initialized**
- Server started but outpost failed to initialize
- Logs show "authentik starting" warnings
- Provider configurations not loaded
4. **Version-specific bugs**
- Version 2024.2.2: Known embedded outpost 404 bug (fixed in later versions)
- Version 2024.8.4: Domain-level forward auth issues with embedded outpost
- Version 2024.10.x: Redirect loop issues
5. **Custom `authentik.web.path` configuration**
- If `authentik.web.path` is changed from default `/`, embedded outpost breaks
- Issue #13504 (March 2025) confirms this current limitation
#### 3. **Forward Auth Modes: forward_single vs forward_domain**
**forward_single (Application Level):**
- Separate authentication per application
- Requires unique proxy provider for each app
- Can apply different access policies per app
- Cookie scoped to specific subdomain
- More granular control
**forward_domain (Domain Level):**
- Single sign-on across all subdomains
- One proxy provider for entire domain
- Same access policy for all apps
- Cookie domain: `.example.com`
- Simpler but less granular
**Known Issue:** Version 2024.8.4 has documented issues with domain-level forward auth (Issue #10848)
**Recommendation:** Use `forward_single` mode for 2024.8.4 (which we're doing) ✅
#### 4. **Correct NPM Configuration**
Research confirms NPM configuration must:
- Proxy `/outpost.goauthentik.io` to `http://authentik-server:9000` (NOT port 9443/9444)
- Enable WebSocket support (critical for auth flow)
- Increase buffer sizes for large headers
- Include proper auth_request directives
---
## Current Configuration Analysis
### ✅ What's Correct
1. **Shared infrastructure** - PostgreSQL and Redis connections working
2. **Memory optimization** - 563MB total (excellent)
3. **Environment variables** - AUTHENTIK_HOST, AUTHENTIK_COOKIE_DOMAIN set correctly
4. **Provider mode** - Using `forward_single` (correct for 2024.8.4)
5. **Provider created** - "Organizr Proxy" exists in Authentik
6. **Application created** - "Organizr" app exists and linked to provider
7. **Outpost assignment** - Provider assigned to embedded outpost
### ⚠️ What's Incorrect/Suspicious
1. **Port mapping confusion:**
```yaml
# stacks/authentik.yml
ports:
- "9000:9000" # Web UI - ✅ Correct
- "9444:9443" # Embedded outpost - ❌ WRONG ASSUMPTION
```
- Port 9443 is not needed for embedded outpost
- Embedded outpost serves on port 9000, not 9443
- This port mapping may be causing confusion but not the root issue
2. **NPM proxy_pass configuration:**
```nginx
# Previous attempt (from session doc)
location /outpost.goauthentik.io {
proxy_pass https://localhost:9444/outpost.goauthentik.io;
# ❌ Wrong port (9444) and wrong protocol (https)
}
```
- Should be: `http://authentik-server:9000/outpost.goauthentik.io`
- Currently reverted, so not in production
3. **Outpost initialization warnings:**
```
{"error":"authentik starting","event":"failed to proxy to backend","level":"warning"}
```
- Repeated many times during container startup
- Suggests embedded outpost may not be fully initializing
- Could be transient startup errors or ongoing issue
### 🧪 Test Results
```bash
# ✅ Ping endpoint works (embedded outpost is running)
$ curl http://192.168.86.149:9000/outpost.goauthentik.io/ping
Status: 204 No Content (empty response body)
# ❌ Auth endpoint returns 404 (provider configuration not loaded)
$ curl http://192.168.86.149:9000/outpost.goauthentik.io/auth/nginx
Status: 404 Not Found
# ❌ Port 9443 internally returns 400 Bad Request
$ docker exec authentik-server python3 -c "import urllib.request; ..."
HTTPError: HTTP Error 400: Bad Request
# ❌ Port 9444 externally expects HTTPS
$ curl http://192.168.86.149:9444/outpost.goauthentik.io/ping
Error: Client sent an HTTP request to an HTTPS server
# ✅ Authentik API accessible
$ curl http://192.168.86.149:9000/api/v3/
Status: 200 OK
```
**Diagnosis:** Embedded outpost is running (ping works) but not serving auth endpoints (404). This indicates the provider configuration is not being loaded by the outpost.
---
## Implementation Strategy
### Option A: Fix Embedded Outpost (PREFERRED - Keep Container Count Low)
**Goal:** Make embedded outpost serve the `/auth/nginx` endpoint correctly
**Approach:**
1. Remove unnecessary port 9444 mapping from docker-compose
2. Update any NPM configs to use port 9000 (not 9444)
3. Investigate why provider isn't loading in embedded outpost:
- Check Authentik admin UI → System → Outposts
- Verify "authentik Embedded Outpost" status
- Check provider assignment
- Review outpost logs for initialization errors
4. Test configuration changes incrementally
5. Monitor outpost initialization after restarts
**Advantages:**
- ✅ Lower container count (preferred requirement)
- ✅ Simpler architecture
- ✅ Less resource usage
- ✅ Fewer moving parts
**Risks:**
- ⚠️ Version 2024.8.4 may have embedded outpost bugs
- ⚠️ Limited documentation for troubleshooting embedded outposts
- ⚠️ May hit version-specific limitations
### Option B: Deploy Standalone Outpost (FALLBACK)
**Goal:** Deploy separate `authentik/proxy` container for forward auth
**Approach:**
1. Create standalone outpost in Authentik UI
2. Generate outpost token
3. Add `authentik-proxy` container to stack
4. Configure to connect to main Authentik server
5. Update NPM to use standalone outpost endpoint
**Advantages:**
- ✅ More reliable (research shows better stability)
- ✅ Better documented in community guides
- ✅ Avoids version-specific embedded outpost issues
- ✅ Cleaner separation of concerns
**Disadvantages:**
- ❌ Additional container (+1 to count)
- ❌ Slightly more complex configuration
- ❌ Additional resource usage (~100-200MB)
**Configuration Example:**
```yaml
authentik-proxy:
image: ghcr.io/goauthentik/proxy:2024.8.4
container_name: authentik-proxy
restart: unless-stopped
environment:
AUTHENTIK_HOST: https://auth.schweitz.net
AUTHENTIK_INSECURE: false
AUTHENTIK_TOKEN: <outpost-token-from-ui>
ports:
- "9443:9443"
networks:
- docker-dataplane
depends_on:
- authentik-server
```
---
## Decision: Try Option A First, Fallback to Option B
**Rationale:**
- User preference: Keep container count low
- Option A aligns with architecture goals
- Option B is a known working solution if A fails
- We have a clear rollback path
**Rollback Point:** Current configuration (Milestone 2 complete)
- Authentik running and healthy
- Google OAuth working
- No forward auth enabled on any services
- All services accessible without SSO
**Rollback Command:**
```bash
# If Option A fails, we can:
# 1. Revert stacks/authentik.yml to current version
# 2. Keep Google OAuth working
# 3. Proceed with Option B (standalone outpost)
```
---
## Next Steps (Option A Implementation)
### Phase 1: Configuration Cleanup
1. Update [stacks/authentik.yml](../../stacks/authentik.yml) - remove port 9444 mapping
2. Verify port 9000 is the only exposed port for Authentik server
3. Redeploy stack and verify containers restart successfully
### Phase 2: Embedded Outpost Investigation
4. Access Authentik admin UI at https://auth.schweitz.net
5. Navigate to System → Outposts → authentik Embedded Outpost
6. Verify status and configuration:
- Status should be "Up" (green)
- Providers should include "Organizr Proxy"
- Last seen timestamp should be recent
7. Check outpost logs for errors
8. Test endpoints again after verification
### Phase 3: NPM Configuration (if outpost working)
9. Update NPM proxy for home.schweitz.net with correct forward auth config
10. Test auth flow: redirect → login → return to app
11. Verify no redirect loops
12. Check cookie persistence
### Phase 4: Documentation & Rollback Prep
13. Document all changes in this session file
14. Update STATUS.md with progress
15. Create backup before each major change
16. Prepare Option B configuration (don't deploy yet)
---
## References
- **Research:** Comprehensive Authentik + NPM implementation guide (see research notes)
- **Official Docs:** https://docs.goauthentik.io/docs/add-secure-apps/providers/proxy/
- **GitHub Issues:**
- #8956: Embedded outpost 404 after 2024.2.2 update
- #10848: Domain-level forward auth issues in 2024.8.4
- #12503: Non-standard port issues
- #13504: Custom web path breaks embedded outpost
---
## Session Status
**Current Phase:** Root cause analysis complete, ready to implement Option A
**Ready to Proceed:** ✅ Yes
- Clear understanding of architecture
- Identified configuration issues
- Implementation plan defined
- Rollback strategy prepared
**Next Action:** Begin Phase 1 - Configuration cleanup
---
## Option A Implementation Results
### Phase 1: Configuration Cleanup ✅ COMPLETE
**Changes Made:**
1. Updated [stacks/authentik.yml](../../stacks/authentik.yml):
- Removed port `9444:9443` mapping
- Updated comments to clarify embedded outpost architecture
- Port 9000 now documented as serving both web UI and embedded outpost
2. Redeployed Authentik containers:
```bash
docker stop authentik-server authentik-worker
docker rm authentik-server authentik-worker
# Redeployed with updated configuration
```
**Test Results:**
```bash
✅ Ping endpoint: http://192.168.86.149:9000/outpost.goauthentik.io/ping → 204 OK
❌ Auth endpoint: http://192.168.86.149:9000/outpost.goauthentik.io/auth/nginx → 404 Not Found
```
**Conclusion:** Port mapping was not the root cause.
---
### Phase 2: Embedded Outpost Investigation ✅ COMPLETE - DEAD END
**Database Investigation:**
1. **Outpost Status:**
```sql
SELECT * FROM authentik_outposts_outpost;
Result:
- UUID: ccf7f82c-b380-4cac-b84c-62e522435410
- Name: authentik Embedded Outpost
- Type: proxy
- Config: authentik_host = https://auth.schweitz.net ✅
```
2. **Provider Assignment:**
```sql
SELECT * FROM authentik_outposts_outpost_providers;
Result:
- Outpost ID: ccf7f82c-b380-4cac-b84c-62e522435410
- Provider ID: 1 ✅
```
3. **Provider Configuration (ISSUE FOUND):**
```sql
SELECT oauth2provider_ptr_id, mode, external_host, cookie_domain
FROM authentik_providers_proxy_proxyprovider;
Initial Result:
- ID: 1
- Mode: forward_single ✅
- External host: https://home.schweitz.net ✅
- Cookie domain: EMPTY ❌ (should be .schweitz.net)
```
**Fix Attempted:**
```sql
UPDATE authentik_providers_proxy_proxyprovider
SET cookie_domain = '.schweitz.net'
WHERE oauth2provider_ptr_id = 1;
-- Restarted containers to apply changes
docker restart authentik-server authentik-worker
```
**Test Results After Fix:**
```bash
❌ Auth endpoint still returns 404
⚠️ Logs continue to show: "failed to proxy to backend" warnings
```
**Root Cause Identified:**
The embedded outpost in Authentik 2024.8.4 is not properly initializing the `/auth/nginx` endpoint despite:
- ✅ Outpost exists and is configured
- ✅ Provider is assigned to outpost
- ✅ Provider configuration is correct (after fix)
- ✅ Environment variables are correct
- ✅ Ping endpoint works (embedded outpost is running)
- ❌ Auth endpoint never exposed (embedded outpost incomplete initialization)
**Log Evidence:**
```json
{"error":"authentik starting","event":"failed to proxy to backend","level":"warning","logger":"authentik.router"}
```
This warning repeats continuously, indicating the embedded outpost backend is not fully starting.
**Conclusion:** This is a **version-specific limitation** of Authentik 2024.8.4 embedded outpost. Research indicated this version has known issues with embedded outposts (Issue #10848). The embedded outpost approach is a **DEAD END**.
---
## Decision: Proceed with Option B - Standalone Outpost
**Rationale:**
1. Embedded outpost not initializing auth endpoint in 2024.8.4
2. Research shows standalone outpost is more reliable
3. We have a clear implementation path
4. Additional container (+1) is acceptable given situation
**Rollback Status:** Current state saved (Milestone 2 complete, no forward auth active)
**Next Steps:** Deploy standalone `authentik-proxy` container with generated token from Authentik UI
---
**Session continues with Option B implementation...**
---
## Option B Implementation Results
### Phase 1: Standalone Outpost Creation ✅ COMPLETE
**Database Operations:**
1. **Created Standalone Outpost:**
```sql
INSERT INTO authentik_outposts_outpost (uuid, name, type, _config, ...)
VALUES (gen_random_uuid(), 'Standalone Proxy Outpost', 'proxy', ...)
Result:
- UUID: 1c2c07d9-91d1-47e2-a92a-08074dac4289
- Name: Standalone Proxy Outpost
- Type: proxy
```
2. **Assigned Provider to Standalone Outpost:**
```sql
INSERT INTO authentik_outposts_outpost_providers (outpost_id, provider_id)
VALUES ('1c2c07d9-91d1-47e2-a92a-08074dac4289', 1)
Result: Provider "Organizr Proxy" now assigned to standalone outpost ✅
```
3. **Generated API Token:**
```sql
INSERT INTO authentik_core_token (identifier, key, ...)
VALUES ('ak-outpost-1c2c07d9-91d1-47e2-a92a-08074dac4289-api',
'bbb141895ac83f0e177857cb16bb9a0d9f082e81e758e6616d25d35c4e2b', ...)
Result: Token created successfully ✅
```
### Phase 2: Container Deployment ✅ COMPLETE
**Initial Deployment (Failed):**
```bash
docker run -d --name authentik-proxy \
-p 9445:9443 \
-e AUTHENTIK_HOST=https://auth.schweitz.net \
-e AUTHENTIK_TOKEN=bbb141895ac83f0e177857cb16bb9a0d9f082e81e758e6616d25d35c4e2b \
ghcr.io/goauthentik/proxy:2024.8.4
Error: Container crash-looping
Cause: "failed to connect to redis" - "dial tcp [::1]:6379: connect: connection refused"
```
**Issue Identified:** Standalone outpost requires Redis configuration (not automatically inherited).
**Fix Applied:**
```bash
docker run -d --name authentik-proxy \
-p 9445:9443 \
-e AUTHENTIK_HOST=https://auth.schweitz.net \
-e AUTHENTIK_HOST_BROWSER=https://auth.schweitz.net \
-e AUTHENTIK_TOKEN=bbb141895ac83f0e177857cb16bb9a0d9f082e81e758e6616d25d35c4e2b \
-e AUTHENTIK_REDIS__HOST=redis-shared \ # ← Added Redis config
-e AUTHENTIK_REDIS__PORT=6379 \
-e AUTHENTIK_REDIS__DB=0 \
--network docker-dataplane \
ghcr.io/goauthentik/proxy:2024.8.4
Result: Container started successfully ✅
```
### Phase 3: Endpoint Testing ✅ COMPLETE
**Test Results:**
```bash
# Ping endpoint (health check)
$ curl -sk https://192.168.86.149:9445/outpost.goauthentik.io/ping
✅ 204 No Content
# Auth endpoint (requires proper nginx headers)
$ curl -sk https://192.168.86.149:9445/outpost.goauthentik.io/auth/nginx
⚠️ 500 Internal Server Error (expected - needs nginx auth_request headers)
# Log message (expected behavior):
"failed to detect a forward URL from nginx"
```
**Analysis:**
The 500 error is **expected and correct**. The auth endpoint requires specific headers from nginx's `auth_request` directive:
- `X-Original-URL` - The URL being accessed
- `X-Forwarded-Proto` - Protocol (http/https)
- `X-Forwarded-Host` - Original host header
- `X-Forwarded-For` - Client IP
When called directly with curl, these headers are missing, so the outpost returns 500. This confirms the outpost is **working correctly** and ready for NPM integration.
### Phase 4: Final Status ✅ SUCCESS
**Deployment Summary:**
```
Containers Running:
- authentik-server: 70d29c3aae92 (healthy) - Port 9000
- authentik-worker: 21a10bb8f1b9 (healthy)
- authentik-proxy: 02a5f67bbe7d (healthy) - Port 9445 → 9443
Memory Usage:
- authentik-server: ~291MB / 512MB (57%)
- authentik-worker: ~272MB / 384MB (71%)
- authentik-proxy: ~150MB / 256MB (58%)
- Total: ~713MB (under 1GB target) ✅
Outpost Configuration:
- Name: Standalone Proxy Outpost
- UUID: 1c2c07d9-91d1-47e2-a92a-08074dac4289
- Provider: Organizr Proxy (forward_single mode)
- External Host: https://home.schweitz.net
- Cookie Domain: .schweitz.net ✅
- Redis: redis-shared:6379/0 ✅
- Status: Running and healthy ✅
```
**Logs (Healthy Output):**
```json
{"event":"Successfully connected websocket","level":"info","logger":"authentik.outpost.ak-ws","outpost":"ccf7f82c-b380-4cac-b84c-62e522435410"}
{"event":"Starting Metrics server","level":"info","listen":"0.0.0.0:9300","logger":"authentik.outpost.metrics"}
{"event":"Starting HTTP server","level":"info","listen":"0.0.0.0:9000","logger":"authentik.outpost.proxyv2"}
{"event":"Starting HTTPS server","level":"info","listen":"0.0.0.0:9443","logger":"authentik.outpost.proxyv2"}
{"event":"Starting authentik outpost","hash":"tagged","level":"info","logger":"authentik.outpost","version":"2024.8.4"}
```
**Conclusion:** Standalone outpost is **fully operational** and ready for NPM forward auth configuration! 🎉
---
## Next Steps: NPM Forward Auth Configuration
Now that the standalone outpost is working, the next phase is to configure Nginx Proxy Manager to use it for forward authentication on home.schweitz.net (Organizr).
### Required NPM Configuration
Add the following to the **Advanced** tab of the `home.schweitz.net` proxy host:
```nginx
# Increase buffer size for large headers from Authentik
proxy_buffers 8 16k;
proxy_buffer_size 32k;
# Forward authentication via standalone outpost
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;
# Capture auth response headers
auth_request_set $auth_cookie $upstream_http_set_cookie;
auth_request_set $authentik_username $upstream_http_x_authentik_username;
auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
auth_request_set $authentik_email $upstream_http_x_authentik_email;
auth_request_set $authentik_name $upstream_http_x_authentik_name;
auth_request_set $authentik_uid $upstream_http_x_authentik_uid;
# Forward auth headers to application
add_header Set-Cookie $auth_cookie;
proxy_set_header X-authentik-username $authentik_username;
proxy_set_header X-authentik-groups $authentik_groups;
proxy_set_header X-authentik-email $authentik_email;
proxy_set_header X-authentik-name $authentik_name;
proxy_set_header X-authentik-uid $authentik_uid;
# Outpost proxy location
location /outpost.goauthentik.io {
proxy_pass https://authentik-proxy:9443/outpost.goauthentik.io;
proxy_set_header Host $host;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
# WebSocket support (if needed)
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
# Signin redirect handler
location @goauthentik_proxy_signin {
internal;
return 302 https://auth.schweitz.net/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
}
```
**Important Notes:**
1. Use `https://authentik-proxy:9443` as the outpost URL (container name, not IP/localhost)
2. Ensure WebSockets are enabled in NPM proxy host settings
3. Test in incognito window to avoid cookie conflicts
### Testing Plan
1. **Access Organizr:** https://home.schweitz.net
2. **Expected Flow:**
- NPM forwards to Authentik for authentication
- Redirects to https://auth.schweitz.net
- Shows login page with Google OAuth button
- After login, returns to https://home.schweitz.net
- Organizr loads successfully
3. **Verify SSO:** Access should persist across browser sessions
4. **Check Logs:** No errors in authentik-proxy logs
---
## Summary: What We Accomplished
### ✅ Completed
1. **Diagnosed embedded outpost failure** - Version 2024.8.4 limitation confirmed
2. **Created standalone outpost** - Database operations via SQL
3. **Generated API token** - Automated token creation
4. **Deployed authentik-proxy container** - Port 9445, with Redis config
5. **Verified outpost functionality** - All endpoints responding correctly
6. **Memory optimization** - Total usage under 1GB (713MB actual)
### 📊 Final Configuration
| Component | Status | Port | Memory | Notes |
|-----------|--------|------|--------|-------|
| authentik-server | ✅ Healthy | 9000 | 291MB | Web UI + API |
| authentik-worker | ✅ Healthy | - | 272MB | Background tasks |
| authentik-proxy | ✅ Healthy | 9445 | 150MB | **Standalone outpost** |
| **Total** | **✅ Operational** | - | **713MB** | Under 1GB target |
### 🔐 Security Tokens
**Standalone Outpost Token:**
```
Identifier: ak-outpost-1c2c07d9-91d1-47e2-a92a-08074dac4289-api
Key: bbb141895ac83f0e177857cb16bb9a0d9f082e81e758e6616d25d35c4e2b
```
### 📝 Files Modified
1. **[stacks/authentik.yml](../../stacks/authentik.yml)** - Added authentik-proxy service (user updated)
2. **[docs/sessions/2025-11-21-authentik-troubleshooting.md](2025-11-21-authentik-troubleshooting.md)** - Complete session log
3. **Database (postgres-shared):**
- New outpost: `Standalone Proxy Outpost`
- Provider assignment updated
- API token created
### 🎯 Milestone Progress
- ✅ **Milestone 1:** Authentik Deployment (Complete)
- ✅ **Milestone 2:** Google OAuth Integration (Complete)
- 🔄 **Milestone 3:** Forward Auth for Organizr (Ready - NPM config needed)
- ⏳ **Milestone 4:** Core API OIDC (Pending)
- ⏳ **Milestone 5:** Remaining Services (Pending)
---
## Lessons Learned
### What Went Well
1. **Systematic troubleshooting approach** - Isolated the issue to embedded outpost
2. **Database-driven configuration** - Created outpost via SQL when UI wasn't clear
3. **Incremental testing** - Caught Redis issue immediately
4. **Research-informed decisions** - Documentation helped identify Redis requirement
### Key Insights
1. **Embedded outpost limitations** - Version 2024.8.4 has known issues, standalone is more reliable
2. **Redis is required** - Standalone outposts need explicit Redis configuration
3. **Auth endpoint behavior** - 500 errors without nginx headers are expected
4. **Memory efficiency** - Standalone outpost uses less memory than embedded (~150MB vs potential overhead)
### For Future Implementations
1. **Start with standalone outposts** - More reliable, easier to troubleshoot
2. **Always check dependencies** - Redis, database connections must be explicit
3. **Test endpoints progressively** - Ping → Auth → Full flow
4. **Use container names** - Not IPs or localhost in Docker networking
---
**Session Status:** ✅ **SUCCESS** - Standalone outpost deployed and operational
**Next Session:** NPM forward auth configuration and SSO testing for Organizr
---
**End of 2025-11-21 Authentik Troubleshooting Session**
docs/sessions/2025-11-23-admin-sso-setup.md
-209
View File
@@ -1,209 +0,0 @@
# Admin-Level SSO Setup Guide
**Date:** 2025-11-23
**Objective:** Create separate user-level and admin-level SSO providers for proper access control
## Overview
This guide sets up a two-tier SSO architecture:
- **User Services Proxy** - For general authenticated access (Organizr)
- **Admin Services Proxy** - For administrative interfaces (Core API, future admin tools)
## Prerequisites
- Authentik accessible at https://auth.schweitz.net
- Admin credentials: akadmin / yzXAhiBAggPB5cz
- Standalone outpost running on port 9445
## Step 1: Create Admin Group
1. Navigate to https://auth.schweitz.net
2. Log in as `akadmin`
3. Go to **Directory** → **Groups**
4. Click **Create**
5. Fill in:
- **Name:** `homelab-admins`
- **Parent:** (none)
- Click **Create**
6. Click on the new `homelab-admins` group
7. Go to **Users** tab
8. Click **Add existing user**
9. Select your user (jpmschweitzer@gmail.com)
10. Click **Add**
## Step 2: Create Admin Authorization Policy
1. Go to **Customization** → **Policies**
2. Click **Create** → **Group Membership Policy**
3. Fill in:
- **Name:** `Admin Group Required`
- **Groups:** Select `homelab-admins`
- Click **Create**
## Step 3: Create Admin Proxy Provider
1. Go to **Applications** → **Providers**
2. Click **Create** → **Proxy Provider**
3. Fill in:
- **Name:** `Admin Services Proxy`
- **Authorization flow:** `default-provider-authorization-implicit-consent`
- **Mode:** `Forward auth (single application)`
- **External host:** `https://api.schweitz.net`
- **Cookie domain:** `.schweitz.net`
- **Token validity:** `hours=8`
- Click **Next**
4. On Policy Bindings page:
- Click **Bind existing policy**
- Select `Admin Group Required`
- **Order:** 0
- Click **Create**
## Step 4: Create Core API Application
1. Go to **Applications** → **Applications**
2. Click **Create**
3. Fill in:
- **Name:** `Core API`
- **Slug:** `core-api`
- **Provider:** Select `Admin Services Proxy`
- **Launch URL:** `https://api.schweitz.net`
- **Policy engine mode:** `all` (require all policies to pass)
- Click **Create**
## Step 5: Assign Provider to Standalone Outpost
1. Go to **Applications** → **Outposts**
2. Click on **Outpost Standalone Proxy Outpost**
3. In the **Applications** field, you should see `Organizr`
4. Add `Core API` to the applications list
5. Click **Update**
6. Wait 10-20 seconds for the outpost to reconnect
7. Check logs: `docker logs authentik-proxy --tail 50`
- Should see: "WebSocket connected" and no errors
## Step 6: Verify NPM Configuration
The NPM config for `api.schweitz.net` should already be correct:
```nginx
# Forward auth to standalone outpost
auth_request /outpost.goauthentik.io/auth/nginx;
# Outpost proxy location
location /outpost.goauthentik.io {
proxy_pass https://localhost:9445/outpost.goauthentik.io;
# ... rest of config
}
```
**No changes needed to NPM** - The outpost automatically handles routing to the correct provider based on the external host.
## Step 7: Test Admin Access
1. **Test in incognito window:**
```bash
# Open incognito window
https://api.schweitz.net/docs
```
2. **Expected flow:**
- Redirects to https://auth.schweitz.net
- Shows Google OAuth login
- After authentication, checks group membership
- If in `homelab-admins` group → allows access
- If NOT in group → shows "Access Denied" or "Insufficient Permissions"
3. **Verify headers are passed:**
```bash
# After logging in, check developer tools → Network → Headers
# Should see X-authentik-groups containing "homelab-admins"
```
## Step 8: Rename Organizr Provider (Optional)
For consistency, rename the existing provider:
1. Go to **Applications** → **Providers**
2. Click on `Organizr Proxy`
3. Change **Name** to `User Services Proxy`
4. Click **Update**
## Architecture Diagram
```
User → https://api.schweitz.net
↓
NPM: Forward auth check
↓
Standalone Outpost (port 9445)
↓
Authentik: Check which provider matches external host
↓
Provider: "Admin Services Proxy" (for api.schweitz.net)
↓
Policy: "Admin Group Required"
↓
✅ User in homelab-admins → Allow
❌ User NOT in group → Deny (403)
```
## Verification Checklist
- [ ] Admin group `homelab-admins` created
- [ ] Your user added to `homelab-admins` group
- [ ] Policy `Admin Group Required` created
- [ ] Provider `Admin Services Proxy` created with policy binding
- [ ] Application `Core API` created and linked to provider
- [ ] Outpost has both `Organizr` and `Core API` applications assigned
- [ ] Outpost logs show successful WebSocket connection
- [ ] Test access to https://api.schweitz.net/docs requires auth
- [ ] After auth, access is granted (user is in admin group)
- [ ] X-authentik-groups header contains `homelab-admins`
## Troubleshooting
### Issue: "Access Denied" even though user is in admin group
**Check:**
```bash
# Verify policy is bound to provider
curl -s -H "Authorization: Bearer 9blMGz71CFMJszs7AedQefgydpTnwvybjmMn0AlYilIKBV5LIq7snqnCodwX" \
https://auth.schweitz.net/api/v3/providers/proxy/ | \
python3 -m json.tool | grep -A 20 "Admin Services"
```
### Issue: Outpost not picking up new provider
**Fix:**
```bash
# Restart outpost
docker restart authentik-proxy
# Check logs
docker logs authentik-proxy --tail 100
```
### Issue: Still using old provider
**Check:**
```bash
# Verify external host is EXACTLY "https://api.schweitz.net" (no trailing slash)
# Authentik matches providers by exact external host match
```
## Next Steps
After admin SSO is working:
1. Mark Milestone 4 as complete in STATUS.md
2. Continue to Milestone 5: Protect remaining services
- git.schweitz.net (Gitea) → Admin provider
- amp.schweitz.net (AMP) → User provider
- tatlock.schweitz.net → User provider
3. Update CHANGELOG.md with 0.8.3-admin-sso version
## Reference
- Authentik Proxy Provider Docs: https://docs.goauthentik.io/docs/providers/proxy/
- Group Policies: https://docs.goauthentik.io/docs/policies/expression/
- Outpost Configuration: https://docs.goauthentik.io/docs/outposts/
docs/sessions/2025-11-23-performance-benchmark.md
-211
View File
@@ -1,211 +0,0 @@
# Core API vs Ollama Direct Performance Benchmark
**Date:** 2025-11-23
**Purpose:** Investigate reported performance differences between Core API and direct Ollama access
## Executive Summary
**TLDR: Core API performance is comparable to direct Ollama (<10% overhead on average)**
### Key Findings
1. ✅ **Non-streaming requests:** Core API shows minimal overhead (0.9% - 6.2%)
2. ✅ **Streaming requests:** Core API is actually faster for first token (-167ms!)
3. ✅ **Resource usage:** Both endpoints use similar CPU/GPU resources
4. ⚠️ **First load latency:** Ollama has ~13s delay on first request (model loading)
## Test Configuration
- **Model:** `gemma:2b` (fast, 2B parameter model)
- **Ollama:** http://192.168.86.149:11434
- **Core API:** http://192.168.86.149:8083
- **Test prompts:** Short (10 tokens), Medium (100 tokens), Long (500 tokens)
- **Runs per test:** 3 iterations
## Benchmark Results
### Non-Streaming Performance
| Test | Ollama Avg | Core API Avg | Overhead | % Difference |
|------|------------|--------------|----------|--------------|
| Short (10 tokens) | 4.780s | 0.347s | -4432ms | **-92.7%** ✓ |
| Medium (100 tokens) | 0.426s | 0.606s | +180ms | **+42.2%** ⚠️ |
| Long (500 tokens) | 3.240s | 3.270s | +30ms | **+0.9%** ✓ |
| **Overall Average** | 2.815s | 1.408s | -1408ms | **-50.0%** ✓ |
**Analysis:**
- Short test shows Ollama had a 13s **model loading delay** on first run
- Excluding warmup, overhead is minimal (0.9% - 6.2%)
- For longer responses (500 tokens), overhead is negligible
### Streaming Performance
| Metric | Ollama Direct | Core API | Difference |
|--------|---------------|----------|------------|
| **Time to First Token** | 0.198s | 0.031s | **-167ms** ✓ |
| **Total Time** | 3.214s | 3.414s | +200ms (+6.2%) |
| **Tokens/Second** | 164.6 | 150.8 | -13.8 tok/s |
**Analysis:**
- Core API delivers first token **167ms faster** (likely caching/optimization)
- Total throughput is 6.2% slower (acceptable for abstraction layer)
- Streaming performance is well within acceptable range
## Resource Usage (Idle State)
```
Container CPU % Memory % of Limit
------------------------------------------------------
ollama 0.07% 703.9MiB / 8GiB 8.59%
core-api 0.48% 504MiB / 2GiB 24.61%
GPU Utilization: 0% (idle)
GPU Memory: 2395 MiB / 11264 MiB (21%)
```
**System State:**
- CPU: 2.1% user, 95.9% idle
- RAM: 9GB / 16GB used (56%)
- Swap: 1.3GB / 2GB used
## Performance Analysis
### Why is Core API Sometimes Faster?
The benchmark shows Core API is often comparable or even faster than direct Ollama. This seems counterintuitive, but here's why:
1. **Efficient FastAPI async handling** - Non-blocking I/O reduces overhead
2. **Minimal middleware** - Only CORS and logging add <10ms
3. **No heavy memory layer active** - Memory system exists but doesn't slow requests
4. **HTTP connection pooling** - httpx AsyncClient reuses connections
5. **Measurement variance** - Network/scheduling jitter affects sub-second measurements
### Where is the 42% Overhead in Medium Test?
The "medium" test showed +180ms overhead:
- Ollama: 0.426s average
- Core API: 0.606s average
**Root cause:** Likely serialization overhead for medium-length responses
- Request parsing: JSON → Pydantic models
- Response formatting: Ollama format → OpenAI format
- SSE streaming setup (even for non-streaming requests)
**Impact:** Acceptable - only affects responses in 100-200 token range
### First Request Latency (13s)
The "short" test Run 1 showed Ollama taking 13.797s:
- This is **model loading time** (cold start)
- Ollama loads model into GPU memory on first request
- Subsequent requests use cached model (0.2-0.3s)
**Not a Core API issue** - both endpoints experience this warmup delay
## Bottleneck Identification
Based on the benchmarks, here are the confirmed bottlenecks:
### ✓ NOT Bottlenecks (Performance is Good)
1. **Core API abstraction layer** - Adds <10% overhead
2. **FastAPI framework** - Efficient async handling
3. **JSON serialization** - Fast enough for this use case
4. **Network hop** (client → Core API → Ollama) - Minimal latency
### ⚠️ Actual Bottlenecks (If You're Experiencing Slowness)
If you're experiencing poor performance, it's likely one of these:
1. **Client-side issues:**
- Network latency to server
- Client HTTP library blocking/synchronous calls
- Browser tab throttling
- Open WebUI buffering/rendering
2. **Model/GPU issues:**
- Model not loaded (13s cold start)
- GPU memory fragmentation
- Other GPU processes competing (AMP, Jellyfin transcoding)
3. **System resources:**
- 9GB RAM used (56%) - some swap pressure
- CPU load from other services (AMP using 27% RAM)
## Recommendations
### For Current Setup (No Changes Needed)
✅ **Core API performance is GOOD** - Keep using it for:
- OpenAI API compatibility
- Open WebUI integration
- Conversation memory features
- Infrastructure automation
### If You Experience Slowness
1. **Check client-side:**
```bash
# Test direct from terminal
time curl -X POST http://192.168.86.149:8083/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "gemma:2b", "messages": [{"role": "user", "content": "Hello"}]}'
```
2. **Monitor GPU usage:**
```bash
watch -n 1 nvidia-smi
# Check if GPU is loaded with other tasks
```
3. **Check if model is loaded:**
```bash
curl http://192.168.86.149:11434/api/tags
# First request after restart takes 13s to load model
```
4. **Reduce concurrent GPU load:**
- Don't use Jellyfin transcoding + AI chat simultaneously
- AMP game servers may use GPU for some tasks
### Optional Optimizations (If Needed)
**For sub-second responses:**
- Use `gemma:2b` instead of `gemma:7b` (3x faster, similar quality)
- Pre-load model: `docker exec ollama ollama run gemma:2b "test"`
**For long conversations:**
- Enable memory tier consolidation (already implemented)
- Use streaming responses for better UX
**For API-heavy workloads:**
- Increase Core API container CPU limit
- Enable response caching for identical requests
## Conclusion
**The Core API is performing excellently.**
- Average overhead: <10%
- Streaming first token: -167ms (faster!)
- Resource usage: Minimal
If you're experiencing slow performance, it's likely:
1. Client-side buffering/rendering (Open WebUI)
2. Cold start model loading (first request)
3. GPU contention with other services
The benchmark proves the abstraction layer is **not** the bottleneck.
## Test Scripts
Benchmark scripts are available at:
- `/tmp/benchmark_ollama_vs_api.py` - Comprehensive non-streaming test
- `/tmp/test_streaming_performance.py` - Streaming performance test
- `/tmp/monitor_resources.sh` - System resource monitoring
To re-run:
```bash
python3 /tmp/benchmark_ollama_vs_api.py
python3 /tmp/test_streaming_performance.py
```
docs/sessions/2025-11-24-vram-optimization-analysis.md
-347
View File
@@ -1,347 +0,0 @@
# VRAM Budget Analysis - Multi-Model Strategy
**Hardware**: RTX 2080 Ti (11GB VRAM)
**Goal**: Keep orchestrator loaded + room for expert models
## Current Model Inventory
| Model | Size on Disk | VRAM When Loaded | Quantization |
|-------|--------------|------------------|--------------|
| **mistral:7b** | 4.4GB | ~5.1GB | Q4_K_M |
| **mistral:7b Q4_K_S** | 4.1GB | ~4.7GB | Q4_K_S |
| **mistral:7b Q3_K_M** | 3.5GB | ~4.0GB | Q3_K_M |
| **codegemma:latest** | 5.0GB | ~5.8GB | Unknown |
| **codestral:latest** | 12GB | ~13GB | Too large! |
## Key Finding: Q3 Removes Tool Support ❌
**Critical Issue**: The Q3_K_M quantization **removes tool calling capability**.
```
mistral:7b Q4_K_M:
Capabilities: completion, tools ✅
mistral:7b Q3_K_M:
Capabilities: completion ❌ No tools!
```
**This means**: You cannot use Q3 for the orchestrator. Tool calling requires Q4 or higher.
---
## Scenario Analysis
### Scenario 1: Current Setup (mistral:7b Q4_K_M)
```
Total VRAM: 11.0 GB
├─ mistral:7b Q4: 5.1 GB (46%) ← Orchestrator (always loaded)
├─ Overhead: 1.2 GB (11%)
└─ Available: 4.7 GB (43%) ← For expert models
```
**What fits in 4.7GB free space:**
- ✅ codegemma:latest (5.8GB) - **Does NOT fit** (need 5.8GB, have 4.7GB)
- ❌ codestral:latest (13GB) - **Does NOT fit** (way too large)
- ✅ gemma3:4b (4.5GB) - **Barely fits** (general purpose)
- ✅ qwen2.5:3b (3.5GB) - **Fits comfortably** (if available)
**Reality Check**: You **cannot** load codegemma or codestral alongside mistral:7b Q4.
---
### Scenario 2: Slightly Smaller Q4 (mistral:7b-instruct-q4_K_S)
```
Total VRAM: 11.0 GB
├─ mistral:7b Q4_K_S: 4.7 GB (43%) ← Orchestrator (slightly smaller)
├─ Overhead: 1.2 GB (11%)
└─ Available: 5.1 GB (46%) ← For expert models
```
**Savings**: 400MB (5.1GB → 4.7GB)
**What fits now:**
- ⚠️ codegemma:latest (5.8GB) - **Still doesn't fit** (need 5.8GB, have 5.1GB)
- ❌ codestral:latest (13GB) - **No chance**
- ✅ gemma3:4b (4.5GB) - **Fits with room to spare**
**Benefit**: Not enough to matter. Still can't fit codegemma.
---
### Scenario 3: Dynamic Loading (Current Ollama Behavior)
**This is what Ollama already does by default!**
```
Step 1: Only orchestrator loaded
├─ mistral:7b Q4: 5.1 GB
├─ Overhead: 1.2 GB
└─ Available: 4.7 GB
Step 2: User requests code generation
├─ Unload mistral:7b (-5.1GB)
├─ Load codestral (+13GB) ← Swaps automatically
└─ Available: 0 GB (codestral fills VRAM)
Step 3: Codestral finishes, times out
├─ Unload codestral (-13GB)
├─ Load mistral:7b (+5.1GB) ← Swaps back
└─ Back to Step 1
```
**How it works:**
- Ollama has a `keep_alive` timer (default: 5 minutes)
- When a model isn't used for 5min, it's unloaded from VRAM
- When you request a different model, Ollama swaps them automatically
**Cold start times:**
- Loading mistral:7b: ~2-3 seconds
- Loading codestral:22b: ~8-10 seconds
- Loading codegemma:9b: ~3-4 seconds
---
## The Math: Why Expert Models Don't Fit
Your 11GB VRAM budget breaks down like this:
```
11GB total VRAM
- 5.1GB orchestrator (mistral:7b Q4)
- 1.2GB system overhead
━━━━━━━━━━━━━━━━━━━━━━
= 4.7GB available
But your expert models need:
- codestral:22b = 13GB ❌ (needs 8GB more than you have)
- codegemma:9b = 5.8GB ❌ (needs 1GB more than available)
```
**Even if you use the smallest possible orchestrator:**
```
11GB total VRAM
- 3.8GB orchestrator (gemma3-tools:1b, unreliable!)
- 1.2GB system overhead
━━━━━━━━━━━━━━━━━━━━━━
= 6.0GB available
Still not enough for:
- codestral:22b = 13GB ❌ (needs 7GB more)
- codegemma:9b = 5.8GB ✅ (fits, but orchestrator is unreliable)
```
---
## Reality: You Need Dynamic Loading
**Conclusion**: With 11GB VRAM, you **cannot** keep both:
1. A reliable orchestrator (min 4.7GB for mistral Q4_K_S)
2. Large expert models (5.8GB+ for code models)
**loaded simultaneously**.
### Option A: Accept Dynamic Loading (Recommended)
**Keep orchestrator loaded** with `keep_alive=-1`, but expert models swap in/out:
```python
# In Core API orchestrator.py
self.llm = ChatOllama(
model="mistral:7b", # Use Q4_K_M or Q4_K_S
keep_alive=-1, # Never unload orchestrator
)
# When calling expert models:
codestral_llm = ChatOllama(
model="codestral:latest",
keep_alive="5m", # Auto-unload after 5 min idle
)
```
**How it works in practice:**
1. **Orchestrator queries** (~80% of requests):
- mistral:7b always in VRAM
- Instant response (~0ms cold start)
- Uses 5.1GB VRAM
2. **Code generation** (~20% of requests):
- mistral:7b stays loaded initially
- Ollama sees codestral request
- **Unloads mistral** automatically
- **Loads codestral** (8-10s cold start)
- Codestral generates code
- After 5min idle: **unloads codestral, reloads mistral**
**Trade-offs:**
- ✅ Orchestrator instant most of the time
- ⚠️ 8-10s cold start when switching to codestral (first code request)
- ⚠️ 2-3s cold start when switching back to orchestrator (after codestral timeout)
- ✅ Can use full-size expert models (codestral:22b, etc.)
---
### Option B: Use Smaller Expert Models
If cold starts are unacceptable, use smaller expert models that fit alongside orchestrator:
```
Orchestrator: mistral:7b Q4_K_S (4.7GB)
Expert: qwen2.5-coder:3b (3.5GB) ← Smaller code model
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total: 8.2GB + 1.2GB overhead = 9.4GB
Available: 1.6GB buffer
```
**Smaller code model options:**
- `qwen2.5-coder:3b` (3.5GB) - Good for simple code tasks
- `starcoder2:3b` (3.2GB) - Focused on code completion
- `deepseek-coder:1.3b` (1.5GB) - Very small, lower quality
**Trade-offs:**
- ✅ Both models always loaded (no cold starts)
- ✅ Instant switching
- ❌ Smaller models = lower code quality
- ❌ Can't use top-tier models like codestral
---
### Option C: Upgrade GPU (Future)
If you want both instant orchestrator AND large expert models:
**RTX 4070 Ti (16GB VRAM):**
```
Total VRAM: 16GB
├─ mistral:7b Q4: 5.1GB (32%)
├─ codestral:22b: 8.0GB (50%) ← Quantized version
├─ Overhead: 1.5GB (9%)
└─ Available: 1.4GB (9%)
```
**With 16GB, you can fit:**
- Orchestrator + codestral Q4 (13GB total)
- Orchestrator + codegemma (11GB total)
- Orchestrator + multiple small experts
---
## Video/Image Models: The Situation
Video and image models are **MUCH larger** than text models:
### Image Generation Models:
- **SDXL (Stable Diffusion XL)**: 6-7GB VRAM
- **Flux.1**: 16-24GB VRAM (dev/schnell variants)
- **SD 1.5**: 3-4GB VRAM (older, lower quality)
### Video Models:
- **AnimateDiff**: 8-12GB VRAM
- **Stable Video Diffusion**: 10-14GB VRAM
- **CogVideoX**: 16-48GB VRAM
### Vision Models (Image Understanding):
- **LLaVA 7B**: 6-7GB VRAM
- **LLaVA 13B**: 10-12GB VRAM
- **GPT-4V equivalent**: 12-16GB VRAM
**Reality Check for 11GB VRAM:**
```
Scenario: Orchestrator + Vision Model
├─ mistral:7b Q4: 5.1GB
├─ LLaVA 7B: 6.5GB
━━━━━━━━━━━━━━━━━━━━━━━
Total needed: 11.6GB ❌ Doesn't fit!
```
Even the smallest vision model (LLaVA 7B) won't fit alongside your orchestrator.
**For image/video generation**: You'd need to fully unload the orchestrator to make room.
---
## Recommendation: Hybrid Strategy
**For your 11GB VRAM constraint, I recommend:**
### 1. Keep Orchestrator Always Loaded
```bash
# mistral:7b Q4_K_M (5.1GB) with keep_alive=-1
# Current setup, no changes needed
```
### 2. Accept Dynamic Loading for Experts
- **Code models**: Load on-demand (codestral, codegemma)
- **Vision models**: Load on-demand (LLaVA)
- **Image gen**: Load on-demand (SDXL)
### 3. Optimize with `keep_alive` Tuning
```python
# Orchestrator: Never unload
orchestrator = ChatOllama(model="mistral:7b", keep_alive=-1)
# Frequently used expert: Keep for 30min
code_expert = ChatOllama(model="codegemma:9b", keep_alive="30m")
# Rarely used expert: Keep for 5min only
vision_expert = ChatOllama(model="llava:7b", keep_alive="5m")
```
**Result:**
- Orchestrator: Always instant
- Frequent code requests: 1st request has 3-4s cold start, then instant for 30min
- Rare vision requests: 6-8s cold start each time
### 4. Monitor and Adjust
Track which expert models you use most:
- If you do a LOT of coding → Keep codegemma loaded longer (`keep_alive="1h"`)
- If coding is rare → Accept the cold start (`keep_alive="5m"`)
---
## Future-Proofing
**If you want to add image/video in the future:**
### Option 1: Offload to CPU (Slow)
```bash
# Run image generation on CPU (very slow, 5-10min per image)
OLLAMA_NUM_GPU=0 ollama run stable-diffusion
```
### Option 2: Dedicated GPU
- Keep RTX 2080 Ti for text models (orchestrator + code)
- Add second GPU for image/video (RTX 3060 12GB, ~$250 used)
### Option 3: Cloud Hybrid
- Local: Text models (orchestrator, code, chat)
- Cloud: Image/video generation (Replicate API, RunPod, etc.)
- Cost: ~$0.002-0.01 per image
---
## Bottom Line
**Your VRAM situation:**
| Capability | Status | Notes |
|------------|--------|-------|
| **Keep orchestrator loaded** | ✅ Yes | 5.1GB with mistral:7b Q4 |
| **+ codegemma simultaneously** | ❌ No | Need 5.8GB, have 4.7GB free |
| **+ codestral simultaneously** | ❌ No | Need 13GB, have 4.7GB free |
| **+ vision model simultaneously** | ❌ No | Need 6GB+, have 4.7GB free |
| **Dynamic loading (swap models)** | ✅ Yes | 2-10s cold starts |
| **Smaller experts simultaneously** | ✅ Maybe | With 3-4GB models only |
**Verdict**:
- ✅ You CAN keep orchestrator always loaded
- ⚠️ You CANNOT keep large experts loaded simultaneously
- ✅ Dynamic loading works fine with acceptable cold start times
- ❌ Image/video models won't fit even with dynamic loading (need GPU upgrade)
**Best approach**: Keep current setup (mistral:7b Q4 always loaded), accept dynamic swapping for expert models. It's what Ollama is designed to do, and 3-8s cold starts are acceptable for occasional expert model use.
docs/sessions/2025-11-24-vram-optimization-strategy.md
-391
View File
@@ -1,391 +0,0 @@
# VRAM Optimization Strategy for Model Orchestration
**Date**: 2025-11-24
**Context**: Multi-model architecture with always-loaded orchestrator + expert models
**Hardware**: RTX 2080 Ti (11GB VRAM)
## Problem Statement
**Goal**: Keep orchestrator model always loaded in VRAM to prevent cold starts, while maximizing VRAM availability for expert models.
**Current State**:
- Orchestrator: `mistral:7b` (5.1GB VRAM)
- Free VRAM: 4.7GB
- Use case: Orchestrator decides → routes to expert models (codestral, etc.)
**Challenge**: mistral:7b consumes 45% of available VRAM, limiting expert model options.
## VRAM Budget Analysis
### Current Configuration
```
Total VRAM: 11.0 GB
├─ mistral:7b: 5.1 GB (46.4%) - Orchestrator
├─ Overhead: 1.2 GB (10.6%) - System/Ollama
└─ Available: 4.7 GB (42.7%) - For expert models
```
### Desired Configuration
```
Total VRAM: 11.0 GB
├─ Orchestrator: ??? GB (minimize)
├─ Expert Model: ??? GB (maximize)
└─ Overhead: 1.2 GB
```
## Solution Options
### Option 1: Accept gemma3-tools:1b Limitations ⚠️
**VRAM Savings**: 3.8GB (5.1GB → 1.3GB)
```
Orchestrator: gemma3-tools:1b (1.3GB)
Free for experts: 8.5GB
```
**Pros**:
- ✅ Massive VRAM savings (74% reduction)
- ✅ Leaves 8.5GB for expert models
- ✅ Can load codestral:22b (full size) + orchestrator simultaneously
**Cons**:
- ❌ 33% tool calling reliability
- ❌ Wrong tool selection
- ❌ Erratic responses (raw JSON output)
- ❌ Poor user experience
**Verdict**: ❌ **Not recommended** - Unreliability hurts more than VRAM savings help
---
### Option 2: Use Smaller Quantization of mistral:7b ✅ RECOMMENDED
Ollama supports multiple quantization levels. You're currently using Q4_K_M, but Q2 or Q3 exist.
**Available Quantizations**:
- Q2_K: ~2.5GB VRAM (70% quality retention, aggressive)
- Q3_K_M: ~3.2GB VRAM (80% quality, good balance)
- Q4_K_M: ~5.1GB VRAM (90% quality, current)
- Q5_K_M: ~6.2GB VRAM (95% quality)
- Q8: ~7.7GB VRAM (99% quality, near full precision)
**Recommended**: Pull `mistral:7b-instruct-q3_K_M`
```bash
# Pull lower quantization
ollama pull mistral:7b-instruct-q3_K_M
# Update Core API config
# services/core-api/.env
AGENT_MODEL=mistral:7b-instruct-q3_K_M
```
**New VRAM Budget**:
```
Orchestrator: mistral:7b Q3_K_M (3.2GB)
Free for experts: 6.6GB
Savings: 1.9GB (37% reduction)
```
**Pros**:
- ✅ 100% tool calling compatibility (same model architecture)
- ✅ 1.9GB VRAM savings
- ✅ Minimal quality loss (80% of full precision is fine for routing)
- ✅ Proven reliability maintained
**Cons**:
- ⚠️ Slightly lower response quality (acceptable for orchestration)
- ⚠️ May need testing to verify tool calling still works
**Verdict**: ✅ **Best option** - Balanced approach
---
### Option 3: Hybrid Orchestrator (Simple Router + mistral:7b) 🔮 ADVANCED
Use a **two-tier routing system**:
1. **Lightweight classifier** (gemma3-tools:1b) - Always loaded
2. **Full orchestrator** (mistral:7b) - Loaded on demand for complex queries
**Architecture**:
```python
# Tier 1: Fast classifier (always loaded)
if query_is_simple(message):
# Direct routing: "list services" → list_services tool
# Load time: 0ms (always in VRAM)
use_simple_router(gemma3-tools:1b)
else:
# Complex routing: multi-tool, reasoning needed
# Load time: ~2s (load mistral:7b)
use_full_orchestrator(mistral:7b)
```
**VRAM Budget**:
```
Tier 1 (always): gemma3-tools:1b (1.3GB)
Tier 2 (on-demand): mistral:7b (5.1GB, loaded when needed)
Free when Tier 1 only: 8.5GB
Free when both loaded: 3.4GB
```
**Pros**:
- ✅ 8.5GB free for expert models most of the time
- ✅ Only loads mistral:7b when truly needed
- ✅ Simple queries stay fast (no model swap)
**Cons**:
- ❌ Complex implementation (need query classifier)
- ❌ 2s latency spike when switching to Tier 2
- ❌ More failure modes (what if Tier 1 misclassifies?)
**Verdict**: 🔮 **Future enhancement** - Interesting but complex
---
### Option 4: Use Different Base Model 🔍 RESEARCH NEEDED
Look for other tool-capable models with better size/quality trade-offs.
**Candidates to research**:
- `qwen2.5:7b-instruct-q3` - Alibaba's model, claimed good tool support
- `llama3.2:3b-instruct` - Meta's latest, check if tool-capable
- `hermes3:3b` - Nous Research, specifically trained for function calling
**Action**: Test these if available in Ollama registry.
---
## Recommended Implementation: Option 2
### Step 1: Pull Q3 Quantization
```bash
# Check if Q3 variant exists
ollama list | grep mistral
# Pull Q3 quantization (if available)
ollama pull mistral:7b-instruct-q3_K_M
# OR manually create Q3 from modelfile
cat > /tmp/mistral-q3.Modelfile << 'EOF'
FROM mistral:7b
PARAMETER quantization Q3_K_M
EOF
ollama create mistral:7b-q3 -f /tmp/mistral-q3.Modelfile
```
### Step 2: Test Tool Calling with Q3
```bash
# Run our test script with Q3 variant
source .venv/bin/activate
python3 << 'PYEOF'
import asyncio
import httpx
async def test():
payload = {
"model": "mistral:7b-q3",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "List all services"}
],
"tools": [{
"type": "function",
"function": {
"name": "list_services",
"description": "List all running services",
"parameters": {"type": "object", "properties": {}}
}
}]
}
async with httpx.AsyncClient(timeout=60) as client:
r = await client.post("http://localhost:11434/api/chat", json=payload)
data = r.json()
message = data.get("message", {})
if "tool_calls" in message:
print("✅ Q3 quantization: Tool calling WORKS")
print(f" Called: {message['tool_calls'][0]['function']['name']}")
else:
print("❌ Q3 quantization: Tool calling BROKEN")
print(f" Response: {message.get('content', '')[:100]}")
asyncio.run(test())
PYEOF
```
### Step 3: Update Core API Configuration
```bash
# services/core-api/.env
AGENT_MODEL=mistral:7b-q3
```
```bash
# Restart core-api to pick up new model
docker restart core-api
```
### Step 4: Verify VRAM Usage
```bash
# Check new VRAM allocation
curl -s http://localhost:11434/api/ps | jq '.models[] | {name, size_vram_gb: (.size_vram / 1024 / 1024 / 1024)}'
```
**Expected Result**:
```json
{
"name": "mistral:7b-q3",
"size_vram_gb": 3.2
}
```
---
## Expert Model Strategy
With ~6.6GB available after Q3 orchestrator, you can now fit:
### Option A: Single Large Expert
```
Orchestrator: mistral:7b-q3 (3.2GB)
Expert: codestral:22b-q2 (6GB)
Total: 9.2GB / 11GB
```
### Option B: Multiple Smaller Experts
```
Orchestrator: mistral:7b-q3 (3.2GB)
Expert 1: codegemma:7b (4GB) - Code generation
Expert 2: gemma3:4b (3GB) - General knowledge
Total: 10.2GB / 11GB (near full capacity)
```
### Option C: Dynamic Loading (Current Behavior)
```
Orchestrator: mistral:7b-q3 (3.2GB) - Always loaded
Expert: Load on demand (6.6GB available)
- codestral for code
- gemma3:12b for general
- Model swaps as needed
```
---
## Advanced: Ollama Keep Alive Configuration
Control how long models stay in VRAM:
```bash
# Keep orchestrator always loaded (never unload)
curl -X POST http://localhost:11434/api/generate \
-d '{
"model": "mistral:7b-q3",
"keep_alive": -1,
"prompt": "warm up"
}'
# Expert models: unload after 5 minutes idle
curl -X POST http://localhost:11434/api/generate \
-d '{
"model": "codestral:latest",
"keep_alive": "5m",
"prompt": "warm up"
}'
```
**Configuration in Core API**:
```python
# services/core-api/src/agent/orchestrator.py
self.llm = ChatOllama(
model=self.settings.agent_model, # mistral:7b-q3
base_url=self.settings.ollama_base_url,
temperature=0.7,
keep_alive=-1, # Never unload orchestrator
)
```
---
## Testing Checklist
Before switching to Q3 quantization:
- [ ] Pull or create Q3 variant
- [ ] Test tool calling functionality
- [ ] Test tool selection accuracy (list_services vs get_service_details)
- [ ] Test multi-tool workflows
- [ ] Compare response quality vs Q4
- [ ] Verify VRAM usage reduction
- [ ] Test with Open WebUI
- [ ] Monitor for any degradation
If Q3 shows issues:
- Try Q4_K_S (slightly smaller than Q4_K_M)
- Fall back to current Q4_K_M if necessary
---
## Alternative Models Research
If mistral Q3 proves insufficient, test these:
### qwen2.5:7b (Alibaba Cloud)
- Similar size to mistral
- Claimed excellent tool calling
- May have Q3/Q4 variants available
```bash
ollama pull qwen2.5:7b-instruct
# Test with our tool calling script
```
### hermes3:3b (Nous Research)
- Specifically trained for function calling
- 3B parameters (smaller than mistral)
- Check Ollama availability
```bash
ollama search hermes3
```
---
## Summary
**Immediate Action**: Pull `mistral:7b` with Q3_K_M quantization
```bash
# Check available quantizations
ollama show mistral:7b --modelfile
# Pull Q3 if available, or create from Q4
ollama pull mistral:7b-instruct-q3_K_M
```
**Expected Outcome**:
- VRAM savings: 1.9GB (5.1GB → 3.2GB)
- Tool calling: Should work (same architecture)
- Quality: 80% of Q4 (acceptable for routing logic)
- Expert model budget: 6.6GB (up from 4.7GB)
**Risk Mitigation**:
- Test thoroughly before production
- Keep Q4 variant as backup
- Monitor for quality degradation
**Long-term**:
- Research newer models (qwen2.5, hermes3)
- Consider hybrid routing if complexity justified
- Revisit when Ollama adds model multiplexing features
---
**Status**: Research complete, awaiting quantization testing
**Next Steps**: User decision on Q3 testing approach
plans/active/phase2-memory-architecture.md
-414
View File
@@ -1,414 +0,0 @@
# Phase 2: Memory Systems Architecture
**Status:** In Progress
**Started:** 2025-11-13
**Phase Goal:** Persistent 3-tier conversation memory with automatic consolidation
## Overview
The memory system provides persistent, intelligent conversation context using a three-tier architecture:
1. **Tier 1 (Working Memory):** Fast in-memory buffer for recent turns
2. **Tier 2 (Short-term):** SQLite database for summarized conversation history
3. **Tier 3 (Long-term):** Qdrant vector store for semantic search across all conversations
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────────┐
│ Chat Endpoint (/v1/chat/completions) │
│ │
│ 1. Accept user message │
│ 2. Retrieve relevant memory from all tiers │
│ 3. Build context: [Tier 1 + Tier 2 + Tier 3 semantic] │
│ 4. Generate response with Ollama │
│ 5. Store new turn in Tier 1 │
│ 6. Trigger consolidation if needed │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Memory Manager │
│ │
│ - Coordinates all 3 tiers │
│ - Handles memory retrieval │
│ - Triggers consolidation │
│ - Manages conversation sessions │
└─────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌──────────────────┐ ┌─────────────────┐ ┌──────────────────┐
│ Tier 1 │ │ Tier 2 │ │ Tier 3 │
│ Buffer Memory │ │ SQLite Summary │ │ Qdrant Vectors │
│ │ │ │ │ │
│ • In-memory dict │ │ • memory.db │ │ • conversation_ │
│ • Last 10 turns │ │ • Summaries │ │ memory │
│ • < 1ms access │ │ • ~10ms access │ │ • Semantic │
│ • Ephemeral │ │ • Persistent │ │ • ~50ms access │
│ • ~5KB RAM │ │ • ~500KB/100 │ │ • ~1KB per turn │
└──────────────────┘ └─────────────────┘ └──────────────────┘
│ │ │
└───────────────────┴────────────────────┘
│
▼
┌─────────────────────────────┐
│ Memory Consolidation │
│ Service │
│ │
│ Triggers: │
│ • Every 10 messages │
│ • Token limit (2000) │
│ • Conversation end │
│ • Explicit save command │
│ │
│ Actions: │
│ • Tier 1 → Tier 2 summary │
│ • Tier 2 → Tier 3 embed │
│ • Prune old Tier 1 data │
└─────────────────────────────┘
```
## Data Structures
### Tier 1: ConversationBufferMemory
```python
{
"conversation_id": "conv_123",
"turns": [
{
"role": "user",
"content": "What is FastAPI?",
"timestamp": "2025-11-13T10:00:00Z",
"turn_number": 1
},
{
"role": "assistant",
"content": "FastAPI is a modern Python web framework...",
"timestamp": "2025-11-13T10:00:02Z",
"turn_number": 2,
"tokens": {"prompt": 15, "completion": 120, "total": 135}
}
],
"metadata": {
"created_at": "2025-11-13T10:00:00Z",
"last_updated": "2025-11-13T10:00:02Z",
"turn_count": 2,
"total_tokens": 135
}
}
```
### Tier 2: SQLite Schema
```sql
-- conversations table
CREATE TABLE conversations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
conversation_id TEXT UNIQUE NOT NULL,
user_id TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
last_message_at TIMESTAMP,
turn_count INTEGER DEFAULT 0,
total_tokens INTEGER DEFAULT 0,
summary TEXT,
status TEXT DEFAULT 'active' -- active, archived, deleted
);
-- conversation_turns table
CREATE TABLE conversation_turns (
id INTEGER PRIMARY KEY AUTOINCREMENT,
conversation_id TEXT NOT NULL,
turn_number INTEGER NOT NULL,
role TEXT NOT NULL, -- user, assistant, system
content TEXT NOT NULL,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
tokens_prompt INTEGER,
tokens_completion INTEGER,
tokens_total INTEGER,
FOREIGN KEY (conversation_id) REFERENCES conversations(conversation_id),
UNIQUE(conversation_id, turn_number)
);
-- conversation_summaries table (for Tier 2 condensed storage)
CREATE TABLE conversation_summaries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
conversation_id TEXT NOT NULL,
summary_text TEXT NOT NULL,
turn_range_start INTEGER NOT NULL,
turn_range_end INTEGER NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
token_count INTEGER,
FOREIGN KEY (conversation_id) REFERENCES conversations(conversation_id)
);
-- Indexes for performance
CREATE INDEX idx_conversation_id ON conversation_turns(conversation_id);
CREATE INDEX idx_timestamp ON conversation_turns(timestamp);
CREATE INDEX idx_summary_conv ON conversation_summaries(conversation_id);
```
### Tier 3: Qdrant Collection Schema
```python
# Collection: conversation_memory
{
"collection_name": "conversation_memory",
"vectors": {
"size": 384, # all-MiniLM-L6-v2 embedding dimension
"distance": "Cosine"
},
"payload_schema": {
"conversation_id": "string",
"turn_number": "integer",
"role": "string",
"content": "text",
"timestamp": "datetime",
"tokens": "integer",
"summary": "text", # Optional condensed version
"tags": ["string"] # e.g., ["question", "code", "technical"]
}
}
```
## Memory Retrieval Flow
### Query: "What did we discuss about FastAPI?"
```python
# 1. Tier 1: Check recent buffer (last 10 turns)
tier1_results = buffer_memory.get_recent_turns(limit=10)
# Returns last 10 turns if they exist
# 2. Tier 2: Check SQLite summaries
tier2_results = sqlite_memory.search_summaries(
conversation_id="conv_123",
query="FastAPI discussion"
)
# Returns summaries containing "FastAPI"
# 3. Tier 3: Semantic search in Qdrant
tier3_results = qdrant_memory.similarity_search(
query="FastAPI discussion",
limit=5,
filter={"conversation_id": "conv_123"}
)
# Returns 5 most semantically similar turns
# 4. Merge and deduplicate
context = merge_memory_results(tier1_results, tier2_results, tier3_results)
# 5. Build prompt with context
prompt = build_prompt_with_memory(
system_message="You are a helpful assistant",
memory_context=context,
user_message="What did we discuss about FastAPI?"
)
```
## Memory Consolidation Logic
### Trigger Conditions
```python
class ConsolidationTrigger:
MESSAGE_COUNT = 10 # Every 10 messages
TOKEN_LIMIT = 2000 # When context > 2000 tokens
CONVERSATION_END = True # End of conversation
EXPLICIT_SAVE = True # User command: "remember this"
TIME_ELAPSED = 3600 # 1 hour idle
```
### Consolidation Process
```python
async def consolidate_memory(conversation_id: str):
"""
Consolidate memory from Tier 1 → Tier 2 → Tier 3
"""
# 1. Get Tier 1 buffer
buffer = tier1_memory.get_buffer(conversation_id)
if len(buffer.turns) >= 10:
# 2. Summarize buffer using lightweight model
summary = await summarize_conversation(
turns=buffer.turns,
model="gemma:7b"
)
# 3. Store summary in Tier 2 (SQLite)
tier2_memory.add_summary(
conversation_id=conversation_id,
summary=summary,
turn_range=(buffer.turns[0].turn_number, buffer.turns[-1].turn_number)
)
# 4. Embed individual turns to Tier 3 (Qdrant)
for turn in buffer.turns:
embedding = await embed_text(turn.content)
tier3_memory.add_turn(
conversation_id=conversation_id,
turn=turn,
embedding=embedding
)
# 5. Prune Tier 1 buffer (keep only last 5 turns)
tier1_memory.prune(conversation_id, keep_last=5)
```
## File Structure
```
services/core-api/src/
├── memory/
│ ├── __init__.py
│ ├── base.py # Base memory classes
│ ├── tier1_buffer.py # ConversationBufferMemory
│ ├── tier2_sqlite.py # ConversationSummaryMemory
│ ├── tier3_qdrant.py # VectorStoreRetrieverMemory
│ ├── manager.py # MemoryManager (coordinates all tiers)
│ ├── consolidation.py # Consolidation service
│ └── schemas.py # Pydantic models
├── api/
│ └── v1/
│ ├── chat.py # Updated with memory integration
│ ├── memory.py # NEW: Memory API endpoints
│ └── schemas.py # Updated with memory schemas
├── models/
│ ├── ollama_client.py # Existing
│ └── embeddings.py # NEW: Embedding model client
└── utils/
└── database.py # NEW: SQLite utilities
```
## API Endpoints (New)
### GET /v1/conversations
List all conversations
### GET /v1/conversations/{conversation_id}
Get conversation details and history
### GET /v1/conversations/{conversation_id}/turns
Get all turns in a conversation
### POST /v1/conversations/{conversation_id}/search
Semantic search within a conversation
### DELETE /v1/conversations/{conversation_id}
Delete/archive a conversation
### POST /v1/conversations/{conversation_id}/consolidate
Manually trigger memory consolidation
## Configuration Updates
```python
# config.py additions
class Settings(BaseSettings):
# ... existing ...
# Memory Configuration
memory_tier1_max_turns: int = 10
memory_tier2_summary_threshold: int = 10
memory_tier3_enabled: bool = True
# SQLite
sqlite_database_path: str = "/app/data/memory.db"
# Qdrant
qdrant_host: str = "qdrant"
qdrant_port: int = 6333
qdrant_collection_conversations: str = "conversation_memory"
qdrant_collection_documents: str = "documents"
qdrant_collection_user_facts: str = "user_facts"
# Embeddings
embedding_model: str = "sentence-transformers/all-MiniLM-L6-v2"
embedding_dimension: int = 384
```
## Dependencies to Add
```txt
# requirements.txt additions
sqlalchemy==2.0.23 # SQLite ORM
qdrant-client==1.7.0 # Qdrant Python client
sentence-transformers==2.2.2 # Embedding models
torch==2.1.0 # PyTorch (for embeddings)
```
## Implementation Phases
### Phase 2.1: Tier 1 (Day 1)
- ✅ Create base memory classes
- ✅ Implement ConversationBufferMemory
- ✅ Add basic memory schemas
- ✅ Test in-memory storage and retrieval
### Phase 2.2: Tier 2 (Day 2)
- ✅ Setup SQLite database
- ✅ Create schema and migrations
- ✅ Implement ConversationSummaryMemory
- ✅ Add summarization using Ollama
- ✅ Test persistence across restarts
### Phase 2.3: Tier 3 (Day 3)
- ✅ Setup Qdrant collections
- ✅ Implement embedding pipeline
- ✅ Implement VectorStoreRetrieverMemory
- ✅ Test semantic search
- ✅ Test Qdrant connectivity
### Phase 2.4: Integration (Day 4)
- ✅ Create MemoryManager
- ✅ Implement consolidation service
- ✅ Update /v1/chat/completions to use memory
- ✅ Add memory API endpoints
- ✅ Test end-to-end flow
### Phase 2.5: Testing & Polish (Day 5)
- ✅ Comprehensive testing
- ✅ Performance optimization
- ✅ Memory leak checks
- ✅ Documentation updates
- ✅ Integration with Open WebUI
## Success Metrics
- **Tier 1 Performance:** < 1ms access time
- **Tier 2 Performance:** < 10ms query time
- **Tier 3 Performance:** < 50ms semantic search
- **Memory Persistence:** 100% across container restarts
- **Context Relevance:** Semantic search returns appropriate results
- **Memory Growth:** Bounded growth with automatic pruning
- **Container Restart:** Conversations resume with full context
## Testing Plan
1. **Unit Tests:**
- Each tier independently
- Consolidation logic
- Memory retrieval
2. **Integration Tests:**
- Full memory flow
- Container restart persistence
- Multi-conversation handling
3. **Performance Tests:**
- 100 conversations
- 1000 turns total
- Memory usage monitoring
- Query performance benchmarks
4. **User Acceptance:**
- Start conversation
- Restart container
- Resume conversation with context
- Ask about past discussions
- Verify relevant recall
---
**Next Step:** Implement Tier 1 (ConversationBufferMemory)
plans/active/security-implementation-plan.md
-1142
View File
File diff suppressed because it is too large Load Diff
plans/active/unified-agent-architecture.md
-451
View File
@@ -1,451 +0,0 @@
# Unified Agent Architecture Plan
**Date:** 2025-11-23
**Objective:** Build a single intelligent agent that handles all tool routing, multi-modal processing, and agentic reasoning internally, exposing one simple chat endpoint to any UI
## Vision
Instead of configuring functions in Open WebUI (or any other UI), the Core API becomes an intelligent orchestrator that:
1. **Accepts simple chat messages** - Just like talking to ChatGPT
2. **Internally routes to specialized tools/models** - Infrastructure management, web search, code execution, etc.
3. **Streams reasoning/thinking** - Shows what it's doing ("Searching the web...", "Querying database...", "Using expert model...")
4. **Returns unified responses** - Combines results from multiple sources transparently
### Benefits
✅ **UI-agnostic** - Works with Open WebUI, CLI, mobile apps, any client
✅ **No configuration needed** - Users just chat naturally
✅ **Transparent reasoning** - See what's happening under the hood
✅ **Tool discovery** - Agent decides when to use tools, not manual triggers
✅ **Multi-modal support** - Handle text, images, code, infrastructure queries
✅ **Expert model routing** - Use small models for simple tasks, large for complex
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────┐
│ User Interface │
│ (Open WebUI, CLI, Mobile App, etc.) │
└──────────────────────┬──────────────────────────────────────┘
│ Simple chat: "Deploy nginx proxy"
↓
┌─────────────────────────────────────────────────────────────┐
│ Core API - Unified Agent │
│ /v1/chat/completions (OpenAI-compatible endpoint) │
└──────────────────────┬──────────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ Agent Orchestrator (PydanticAI) │
│ ┌──────────────────────────────────────────────┐ │
│ │ Reasoning Loop: │ │
│ │ 1. Analyze user intent │ │
│ │ 2. Select appropriate tool(s) │ │
│ │ 3. Execute tool calls │ │
│ │ 4. Synthesize results │ │
│ │ 5. Stream thinking/reasoning │ │
│ └──────────────────────────────────────────────┘ │
└──────────────────────┬──────────────────────────────────────┘
│
┌──────────────┼──────────────┬──────────────┐
│ │ │ │
↓ ↓ ↓ ↓
┌──────────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐
│ Tool Catalog │ │ Models │ │ Memory │ │ Knowledge │
│ │ │ │ │ │ │ │
│ • Infra Mgmt │ │ • Gemma │ │ • Qdrant │ │ • Web Search │
│ • Web Scrape │ │ • Codestral│ │ • Buffer│ │ • Docs │
│ • File Ops │ │ • Mistral│ │ │ │ │
│ • Code Exec │ │ │ │ │ │ │
└──────────────┘ └──────────┘ └──────────┘ └──────────────┘
```
## Implementation Options
### Option 1: PydanticAI (Current Implementation)
**Pros:**
- Type-safe tool definitions with Pydantic models
- Built-in streaming support with structured output
- Native Ollama integration via HTTP API
- Lightweight and minimal dependencies
- Clear separation of concerns with dependency injection
- Excellent debugging with structured validation
**Cons:**
- Relatively new framework (less established patterns)
- Manual agent loop implementation required
- Less built-in state management compared to stateful frameworks
**Example flow:**
```python
from pydantic_ai import Agent, RunContext
from pydantic import BaseModel
class DeployServiceParams(BaseModel):
service_name: str
compose_yaml: str
class WebSearchParams(BaseModel):
query: str
agent = Agent(
model="ollama:mistral-tools:7b",
system_prompt="You are a homelab infrastructure assistant...",
result_type=str
)
@agent.tool
async def deploy_service(ctx: RunContext[None], params: DeployServiceParams) -> str:
"""Deploy a containerized service via Portainer"""
return await portainer_client.deploy_stack(
params.service_name,
params.compose_yaml
)
@agent.tool
async def web_search(ctx: RunContext[None], params: WebSearchParams) -> str:
"""Search the web and extract content"""
return await scraper.scrape(params.query)
# Streaming with reasoning
async with agent.run_stream(user_message) as stream:
async for chunk in stream.stream_text():
if chunk.type == "tool_call":
yield f"data: {json.dumps({'tool': chunk.tool_name})}\n\n"
elif chunk.type == "text":
yield f"data: {json.dumps({'content': chunk.content})}\n\n"
```
### Option 2: Custom Agent Loop
**Pros:**
- Full control over behavior
- Minimal dependencies
- Optimized for specific use case
- Easier to debug
**Cons:**
- More code to maintain
- Need to implement tool calling protocol
- Reinventing some wheels
- Manual type validation
**Example flow:**
```python
class UnifiedAgent:
def __init__(self):
self.tools = ToolCatalog()
self.model = OllamaClient()
async def process(self, user_message: str):
# 1. Intent analysis
yield {"type": "thinking", "content": "Analyzing your request..."}
intent = await self.analyze_intent(user_message)
# 2. Tool selection
if intent.requires_tool:
yield {"type": "thinking", "content": f"Using {intent.tool_name}..."}
tool_result = await self.tools.execute(intent.tool_name, intent.params)
# 3. Response generation
yield {"type": "thinking", "content": "Generating response..."}
response = await self.model.generate(context=tool_result)
yield {"type": "content", "content": response}
```
### Option 3: Hybrid (PydanticAI + Custom Extensions)
Use PydanticAI's agent framework with custom enhancements:
- Leverage type-safe tool definitions
- Add custom routing logic for multi-model selection
- Enhanced streaming control for reasoning output
- Custom dependency injection for context management
## Recommended Approach: PydanticAI with Custom Extensions
**Phase 1: Core Agent (Week 1)**
- Set up PydanticAI agent with basic tools
- Implement streaming with reasoning output
- Wire up existing infrastructure tools
- Test with simple queries
**Phase 2: Advanced Routing (Week 2)**
- Multi-model routing (small for simple, large for complex)
- Parallel tool execution via async tools
- Error handling and retries with custom logic
- Context management using RunContext
**Phase 3: Multi-Modal (Week 3)**
- Image analysis (if needed)
- Code execution sandbox
- File operations
- Database queries
## Tool Catalog Design
### Tier 1: Infrastructure Tools (Existing)
```python
from pydantic_ai import Agent, RunContext
from pydantic import BaseModel
class ListServicesResult(BaseModel):
services: List[Dict]
class DeployServiceParams(BaseModel):
name: str
compose: str
@agent.tool
async def list_services(ctx: RunContext[None]) -> ListServicesResult:
"""List all running Docker services"""
services = await portainer_client.list_containers()
return ListServicesResult(services=services)
@agent.tool
async def deploy_service(ctx: RunContext[None], params: DeployServiceParams) -> str:
"""Deploy a new service from Docker Compose YAML"""
return await portainer_client.deploy_stack(params.name, params.compose)
@agent.tool
async def create_proxy(ctx: RunContext[None], domain: str, target: str) -> str:
"""Create Nginx reverse proxy for a service"""
return await npm_client.create_proxy_host(domain, target)
@agent.tool
async def check_service_health(ctx: RunContext[None], service: str) -> Dict:
"""Check if a service is healthy"""
return await kuma_client.get_monitor_status(service)
```
### Tier 2: Knowledge Tools
```python
@agent.tool
async def web_search(ctx: RunContext[None], query: str) -> str:
"""Search the web and extract main content"""
return await scraper.scrape_url(query)
@agent.tool
async def query_memory(ctx: RunContext[None], question: str) -> List[str]:
"""Search conversation history for relevant context"""
return await memory.semantic_search(question)
@agent.tool
async def read_documentation(ctx: RunContext[None], topic: str) -> str:
"""Read project documentation"""
docs_path = f"/docs/{topic}.md"
return read_file(docs_path)
```
### Tier 3: Execution Tools (Future)
```python
@agent.tool
async def execute_python(ctx: RunContext[None], code: str) -> str:
"""Execute Python code in sandbox"""
# Future: Integrate code interpreter
pass
@agent.tool
async def query_database(ctx: RunContext[None], sql: str) -> List[Dict]:
"""Query PostgreSQL database"""
# Future: Safe SQL execution
pass
```
## Streaming Reasoning Output
### SSE Format for Transparency
```python
# Stream format
{
"type": "thinking", # or "tool_call", "content", "error"
"content": "Searching the web for nginx configuration...",
"tool": "web_search", # optional, if type is tool_call
"model": "mistral-tools:7b" # optional, which model is being used
}
# Example stream
data: {"type": "thinking", "content": "Analyzing your request..."}
data: {"type": "thinking", "content": "Detected infrastructure task"}
data: {"type": "tool_call", "tool": "list_services", "content": "Checking current services..."}
data: {"type": "thinking", "content": "Found 22 running services"}
data: {"type": "thinking", "content": "Using expert model for response..."}
data: {"type": "model_switch", "from": "gemma:2b", "to": "mistral:7b"}
data: {"type": "content", "content": "Here are your running services:\n\n..."}
data: [DONE]
```
### Open WebUI Integration
Open WebUI already supports streaming, we just need to format it correctly:
```javascript
// Open WebUI will render thinking/reasoning in a collapsible section
// Standard content renders as usual
```
## Model Routing Strategy
### Intent-Based Routing
```python
from pydantic_ai import Agent
class ModelRouter:
MODELS = {
"simple": "ollama:gemma:2b", # Fast, <100 tokens
"general": "ollama:gemma:7b", # Balanced
"expert": "ollama:mistral:7b", # Complex reasoning
"code": "ollama:codestral:latest" # Code tasks
}
async def select_model(self, message: str, context: str) -> str:
# Use lightweight model for routing decision
routing_agent = Agent(
model="ollama:gemma:2b",
result_type=str,
system_prompt="""Analyze this request and categorize:
Categories:
- simple: Greetings, basic facts, short answers
- general: Normal conversation, explanations
- expert: Complex reasoning, multi-step problems
- code: Programming tasks, debugging
Return ONLY the category.
"""
)
result = await routing_agent.run(f"User: {message}\nContext: {context}")
return self.MODELS[result.data.strip()]
```
## Next Steps
1. **Enhance PydanticAI agent** (2-3 hours)
- Add more infrastructure tools
- Improve streaming with reasoning output
- Test with complex queries
2. **Integrate remaining tools** (3-4 hours)
- Migrate all infrastructure controller tools
- Add web scraper tool improvements
- Test multi-tool workflows
3. **Model routing enhancements** (2 hours)
- Refine intent analysis
- Add model selection metrics
- Test performance improvements
4. **Production hardening** (2 hours)
- Enhanced error handling
- Rate limiting
- Logging and monitoring
- Update API documentation
**Total effort:** ~12-15 hours (1-2 weeks of focused work)
## Success Criteria
✅ User can chat naturally without configuring functions
✅ Agent automatically uses tools when appropriate
✅ Streaming shows what the agent is doing
✅ Works with Open WebUI without changes
✅ Can be used from CLI/API directly
✅ Performance is acceptable (<5s for tool-using responses)
✅ Errors are handled gracefully
## Example User Flows
### Flow 1: Infrastructure Query
```
User: "What services are currently running?"
[Thinking: Analyzing request...]
[Thinking: Detected infrastructure query]
[Tool Call: list_services - Fetching service list...]
[Thinking: Processing results...]
[Content: You have 22 services running:
- ollama (healthy)
- core-api (healthy)
- ...]
```
### Flow 2: Complex Task
```
User: "Deploy an nginx proxy for my new blog at blog.schweitz.net"
[Thinking: Breaking down the task...]
[Thinking: Need to deploy nginx and configure NPM]
[Tool Call: deploy_service - Deploying nginx container...]
[Tool Call: create_proxy - Creating proxy host...]
[Thinking: Configuring SSL certificate...]
[Content: Done! Your blog is now accessible at https://blog.schweitz.net
- Nginx container: running
- SSL certificate: active
- Health check: passing]
```
### Flow 3: Knowledge Query
```
User: "How do I configure Headscale?"
[Thinking: Checking documentation...]
[Tool Call: read_documentation(headscale)]
[Thinking: Extracting relevant steps...]
[Content: To configure Headscale on tower-of-joy:
1. Create a user: `headscale users create homelab`
2. Generate auth key: `headscale preauthkeys create...`
...]
```
## Technology Stack
- **Agent Framework:** PydanticAI
- **LLM Integration:** Native Ollama SDK (HTTP API)
- **Tool Framework:** PydanticAI Tools with Pydantic validation
- **Streaming:** SSE (Server-Sent Events)
- **State Management:** RunContext dependency injection
- **Memory:** Existing Qdrant integration
## Risk Mitigation
**Risk:** PydanticAI is relatively new
- **Mitigation:** Strong typing provides safety, active development community
**Risk:** Tool calling may be slow
- **Mitigation:** Async tools enable parallel execution, caching, optimized tools
**Risk:** Reasoning output may be verbose
- **Mitigation:** Configurable verbosity, collapsible UI elements
**Risk:** May not work with all UIs
- **Mitigation:** Stick to OpenAI-compatible streaming format
## Open Questions
1. Should we support function calling format for backwards compatibility?
2. How verbose should reasoning output be?
3. Should we cache tool results?
4. Do we need user confirmation for destructive operations?
5. Should tools have permission levels based on user?
---
**Ready to implement:** Yes ✓
**Estimated timeline:** 1-2 weeks
**Priority:** High (enables true agentic behavior)
plans/completed/architecture-research.md
-578
View File
@@ -1,578 +0,0 @@
# Home Server Container Platform Research
> Research Date: 2025-11-11
> System: tower-of-joy (Zorin OS 16.3, Intel i7-6700, 16GB RAM, RTX 2080 Ti)
## Executive Summary
This document contains comprehensive research on open-source home server solutions for containerizing applications, web servers, file servers, Jellyfin media server, and cloud services like Nextcloud. The research evaluates platforms based on our specific hardware constraints and requirements.
### System Context
**Current Configuration:**
- **OS**: Zorin OS 16.3 (Ubuntu 20.04 based)
- **CPU**: Intel i7-6700 (4 cores, 8 threads, 3.40GHz)
- **RAM**: 16 GB
- **Storage**: 481 GB (365 GB available) - **LIMITED**
- **GPU**: NVIDIA RTX 2080 Ti (11GB VRAM) - **EXCELLENT for transcoding**
- **Docker**: 28.1.1 (already installed)
- **User**: jpmschweitzer
- **Hostname**: tower-of-joy
**Critical Constraints:**
1. Limited storage (481GB) - Rules out storage-intensive solutions
2. Existing OS installation - Prefer solutions that don't require fresh install
3. RTX 2080 Ti excellent for Jellyfin hardware transcoding
4. Docker already installed - Should leverage existing infrastructure
### Requirements
1. **Container orchestration** for running:
- Jellyfin media server (with GPU hardware transcoding)
- Nextcloud (cloud storage with external access)
- File servers
- Web servers
- Various other containerized applications
2. **Web-based management interface** for container/service management
3. **NAS capabilities** (file storage and sharing)
4. **Software-defined networking** - Specifically Tailscale's OSS version (Headscale) or similar
5. **External access capabilities** (secure remote access)
6. **Easy extensibility** for adding more services
7. **GPU passthrough support** for Jellyfin hardware transcoding
---
## Solutions Evaluated
### 1. Portainer + Docker Compose ⭐ **RECOMMENDED**
**Overview:**
Portainer provides a web-based management interface for Docker, allowing you to manage containers, stacks, images, and volumes through an intuitive UI. Combined with Docker Compose for multi-container orchestration.
**Installation Compatibility:**
- ✅ **WORKS ON EXISTING UBUNTU/ZORIN OS**
- No fresh install required
- Installs as a Docker container itself
#### Ratings
| Category | Rating | Notes |
|----------|--------|-------|
| Web UI Quality | ⭐⭐⭐⭐⭐ (9.6/10) | Intuitive dashboard, visual management, real-time monitoring |
| Container/Docker Support | ⭐⭐⭐⭐⭐ | Native Docker integration, full Compose support, stack management |
| GPU Passthrough | ⭐⭐⭐⭐ (4/5) | Full NVIDIA support via Container Toolkit, GPU toggle in UI |
| NAS/File Sharing | ⭐⭐⭐ (3/5) | Not built-in, easily added via Samba/NFS containers |
| Headscale Integration | ⭐⭐⭐⭐⭐ | Excellent - both available as Docker containers |
| Hardware Requirements | ⭐⭐⭐⭐⭐ | Minimal - perfect for 481GB storage constraint |
| Learning Curve | ⭐⭐⭐⭐⭐ (EASY) | Rated 9.6/10 for ease of use, visual interface |
| Community & Ecosystem | ⭐⭐⭐⭐⭐ | Massive Docker ecosystem, active community |
| Extensibility | ⭐⭐⭐⭐⭐ | Add any Docker container via UI, custom stacks |
#### GPU Configuration Example
```yaml
version: '3'
services:
jellyfin:
image: jellyfin/jellyfin:latest
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=all
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
```
#### Pros & Cons
**PROS:**
- ✅ Works on existing OS (no reinstall)
- ✅ Minimal resource footprint (~200MB disk, <100MB RAM for Portainer)
- ✅ Extremely easy to use (9.6/10 rating)
- ✅ Full GPU support for Jellyfin
- ✅ Already have Docker installed
- ✅ Huge ecosystem of containers
- ✅ Perfect for limited storage (481GB)
- ✅ Quick setup (15-30 minutes)
- ✅ Free and open source
- ✅ Excellent for Jellyfin + Nextcloud + file servers
**CONS:**
- ❌ NAS features require separate containers (not integrated)
- ❌ No built-in RAID or advanced storage management
- ❌ Less comprehensive than full NAS solutions
- ❌ File sharing requires additional configuration
#### Expected Challenges
1. Setting up NVIDIA Container Toolkit (one-time setup)
2. Configuring proper GPU permissions
3. Learning Docker Compose syntax (minimal if using UI)
4. Setting up reverse proxy for external access (Nginx/Caddy)
---
### 2. CasaOS - **BEST ALTERNATIVE**
**Overview:**
CasaOS is a beautiful, app-store-like home server operating system that runs on top of existing Linux installations. Designed specifically for home users who want simplicity.
**Installation Compatibility:**
- ✅ **INSTALLS ON EXISTING UBUNTU/ZORIN OS**
- Single curl command: `curl -fsSL https://get.casaos.io | bash`
- Auto-installs Docker if not present
#### Ratings
| Category | Rating | Notes |
|----------|--------|-------|
| Web UI Quality | ⭐⭐⭐⭐⭐ (5/5) | Most elegant UI, app store paradigm, built-in file manager |
| Container/Docker Support | ⭐⭐⭐⭐⭐ | Built on Docker, app store, recognizes existing containers |
| GPU Passthrough | ⭐⭐⭐⭐ (4/5) | NVIDIA support via environment variables |
| NAS/File Sharing | ⭐⭐⭐⭐ (4/5) | Built-in file manager, easy network sharing |
| Headscale Integration | ⭐⭐⭐⭐⭐ | Can install via Docker containers |
| Hardware Requirements | ⭐⭐⭐⭐⭐ | Very light (~500MB for CasaOS) |
| Learning Curve | ⭐⭐⭐⭐⭐ (EASIEST) | Absolute easiest solution, "click and go" |
| Community & Ecosystem | ⭐⭐⭐⭐ (4/5) | Growing community, Docker ecosystem access |
| Extensibility | ⭐⭐⭐⭐⭐ | Full Docker ecosystem, custom app import |
#### Pros & Cons
**PROS:**
- ✅ Installs on existing OS
- ✅ Absolutely beautiful UI
- ✅ Easiest to use (perfect for beginners)
- ✅ App store paradigm
- ✅ Built-in file management
- ✅ GPU support for Jellyfin
- ✅ Minimal resources
- ✅ One-command install
- ✅ Can combine with Portainer
**CONS:**
- ❌ Less granular control than Portainer
- ❌ Newer/smaller community
- ❌ May abstract away some Docker details
- ❌ Advanced features require custom Docker configs
---
### 3. Cockpit + Podman
**Overview:**
Cockpit is a web-based Linux server management tool with a Podman extension for container management. Podman is a daemonless Docker alternative.
**Installation Compatibility:**
- ✅ Works on existing Ubuntu
- Installs via apt package manager
#### Ratings
| Category | Rating | Notes |
|----------|--------|-------|
| Web UI Quality | ⭐⭐⭐⭐ (4/5) | Clean, functional, less polished than alternatives |
| Container/Docker Support | ⭐⭐⭐ (3/5) | Uses Podman (not Docker), compatibility issues |
| GPU Passthrough | ⭐⭐⭐⭐ (4/5) | NVIDIA support with Podman |
| NAS/File Sharing | ⭐⭐ (2/5) | No built-in features |
| Headscale Integration | ⭐⭐⭐⭐ | Available as Podman containers |
| Hardware Requirements | ⭐⭐⭐⭐⭐ | Very lightweight |
| Learning Curve | ⭐⭐⭐ (3/5 - MODERATE) | Requires learning Podman differences |
| Community & Ecosystem | ⭐⭐⭐ (3/5) | Growing, smaller than Docker |
| Extensibility | ⭐⭐⭐ (3/5) | Limited compared to Docker |
**Why Not Recommended:**
- Not compatible with existing Docker setup
- Smaller container ecosystem
- Would require migration from Docker to Podman
- Less intuitive than alternatives
---
### 4. K3s / MicroK8s (Lightweight Kubernetes)
**Overview:**
Lightweight Kubernetes distributions designed for edge computing and resource-constrained environments.
**Installation Compatibility:**
- ✅ Works on existing Ubuntu
- k3s: Single binary installation
- MicroK8s: Snap package (Ubuntu native)
#### Ratings
| Category | Rating | Notes |
|----------|--------|-------|
| Web UI Quality | ⭐⭐⭐ (3/5) | Less intuitive than Portainer |
| Container/Docker Support | ⭐⭐⭐⭐ (4/5) | Uses containerd, complex deployment |
| GPU Passthrough | ⭐⭐⭐⭐⭐ | Excellent GPU support, NVIDIA operator |
| NAS/File Sharing | ⭐⭐ (2/5) | No built-in features |
| Headscale Integration | ⭐⭐⭐⭐ | Can run as pods |
| Hardware Requirements | ⭐⭐⭐⭐ | 150-600MB RAM depending on distro |
| Learning Curve | ⭐ (1/5 - STEEP) | Very steep, Kubernetes concepts required |
| Community & Ecosystem | ⭐⭐⭐⭐⭐ | Massive Kubernetes ecosystem |
| Extensibility | ⭐⭐⭐⭐⭐ | Unlimited, enterprise-grade |
**Why Not Recommended:**
- Massive overkill for home server
- Steep learning curve (weeks to months)
- Complex for simple tasks
- Use case doesn't need Kubernetes orchestration
- More resource overhead than needed
---
### 5. TrueNAS Scale
**Overview:**
Enterprise-grade NAS operating system based on Debian with built-in Kubernetes (K3s) for app deployment.
**Installation Compatibility:**
- ❌ **REQUIRES FRESH INSTALL**
- Not dual-boot friendly
- Requires entire disk
- Minimum 2 disks for storage functionality
#### Ratings
| Category | Rating | Notes |
|----------|--------|-------|
| Web UI Quality | ⭐⭐⭐⭐⭐ | Excellent, comprehensive |
| Container/Docker Support | ⭐⭐⭐ (3/5) | Uses K3s, more complex than Docker |
| GPU Passthrough | ⭐⭐⭐⭐ (4/5) | NVIDIA support in 24.10+, some RTX issues reported |
| NAS/File Sharing | ⭐⭐⭐⭐⭐ | Best-in-class, ZFS, snapshots, replication |
| Headscale Integration | ⭐⭐⭐ | Can deploy as K3s apps |
| Hardware Requirements | ⭐⭐ (2/5) | Requires 2+ disks, storage-intensive |
| Learning Curve | ⭐⭐⭐ (3/5 - MODERATE) | Storage concepts to learn |
| Community & Ecosystem | ⭐⭐⭐⭐⭐ | Large community, enterprise backing |
| Extensibility | ⭐⭐⭐⭐ | App catalog, K3s apps |
**Why Not Recommended:**
- ❌ **REQUIRES FRESH INSTALL** (major dealbreaker)
- ❌ Needs 2+ disks (we have 1)
- ❌ 481GB too small for NAS + apps
- ❌ Overkill for our needs
- ❌ Would lose existing Zorin OS setup
- ❌ Not suitable for our hardware configuration
---
### 6. Unraid
**Overview:**
Popular NAS-focused OS with excellent Docker support and user-friendly interface. Known for flexible storage and parity protection.
**Installation Compatibility:**
- ❌ **REQUIRES FRESH INSTALL**
- Boots from USB drive
- Takes over entire system
#### Ratings
| Category | Rating | Notes |
|----------|--------|-------|
| Web UI Quality | ⭐⭐⭐⭐⭐ | Excellent, polished |
| Container/Docker Support | ⭐⭐⭐⭐⭐ | Native Docker, Community Applications |
| GPU Passthrough | ⭐⭐⭐⭐⭐ | Excellent NVIDIA/AMD support |
| NAS/File Sharing | ⭐⭐⭐⭐⭐ | Excellent, flexible array, parity protection |
| Headscale Integration | ⭐⭐⭐⭐⭐ | Community containers, well-documented |
| Hardware Requirements | ⭐⭐⭐ (3/5) | Works with single disk, benefits from multiple |
| Learning Curve | ⭐⭐⭐⭐ (4/5 - EASY) | Very user-friendly |
| Community & Ecosystem | ⭐⭐⭐⭐⭐ | Massive community, active forums |
| Extensibility | ⭐⭐⭐⭐⭐ | Docker, VMs, plugins |
**Why Not Recommended (Currently):**
- ❌ **REQUIRES FRESH INSTALL** (dealbreaker)
- ❌ **NOT FREE** ($59-$129 license)
- ❌ Would lose existing setup
- ❌ Limited by 481GB storage
- ❌ Boots from USB (uses a port)
**Note:** Best all-in-one solution if starting fresh with more storage. Consider for future rebuild.
---
### 7. Proxmox VE
**Overview:**
Enterprise virtualization platform supporting VMs and LXC containers. Industry-standard for homelabs.
**Installation Compatibility:**
- ❌ **REQUIRES FRESH INSTALL** (typically)
- Can migrate existing Ubuntu to VM (complex)
#### Ratings
| Category | Rating | Notes |
|----------|--------|-------|
| Web UI Quality | ⭐⭐⭐⭐⭐ | Professional, comprehensive |
| Container/Docker Support | ⭐⭐⭐ (3/5) | LXC containers, not Docker directly |
| GPU Passthrough | ⭐⭐⭐⭐⭐ | Excellent, well-documented |
| NAS/File Sharing | ⭐⭐ (2/5) | No built-in, deploy as VM |
| Headscale Integration | ⭐⭐⭐ | Can run in containers/VMs |
| Hardware Requirements | ⭐⭐⭐ (3/5) | Virtualization overhead, 481GB limiting |
| Learning Curve | ⭐⭐ (2/5 - STEEP) | Virtualization concepts required |
| Community & Ecosystem | ⭐⭐⭐⭐⭐ | Huge community, enterprise support |
| Extensibility | ⭐⭐⭐⭐⭐ | Maximum flexibility |
**Why Not Recommended:**
- ❌ Requires fresh install
- ❌ Overkill for our needs
- ❌ Virtualization overhead
- ❌ More complex than needed
- ❌ Limited by 481GB storage
- ❌ Not optimized for Docker
---
### 8. YunoHost
**Overview:**
Debian-based server OS focused on simplifying self-hosting with pre-packaged applications.
**Installation Compatibility:**
- ⚠️ Prefers fresh install
- Can work on existing Debian/Ubuntu (risky)
- May conflict with existing setup
#### Ratings
| Category | Rating | Notes |
|----------|--------|-------|
| Web UI Quality | ⭐⭐⭐⭐ | Good application-focused UI |
| Container/Docker Support | ⭐⭐ (2/5) | Docker support experimental/unofficial |
| GPU Passthrough | ⭐ (1/5) | No specific support |
| NAS/File Sharing | ⭐⭐⭐ | Basic file sharing |
| Headscale Integration | ⭐⭐ | Would require manual setup |
| Hardware Requirements | ⭐⭐⭐⭐ | Lightweight |
| Learning Curve | ⭐⭐⭐⭐ | Easy for app installation |
| Community & Ecosystem | ⭐⭐⭐ | Active, limited app catalog |
| Extensibility | ⭐⭐ | Limited to YunoHost apps |
**Why Not Recommended:**
- ❌ Poor Docker support
- ❌ No GPU support
- ❌ Not suitable for Jellyfin + Docker setup
- ❌ Limited extensibility
- ❌ Prefers fresh install
---
## Software-Defined Networking Solutions
### Headscale ⭐ **RECOMMENDED**
**Overview:**
Open-source, self-hosted implementation of Tailscale control server. Fully compatible with Tailscale clients.
**Key Features:**
- Self-hosted control plane
- Use official Tailscale clients
- ACL support
- Pre-authenticated keys
- Docker container available (`headscale/headscale`)
**Integration:**
- ✅ Excellent Docker integration
- Docker Compose deployment
- Can share network to other containers
- Well-documented setup
**PROS:**
- ✅ Fully self-hosted
- ✅ No external dependencies
- ✅ Uses Tailscale clients
- ✅ Free and open source
- ✅ Active development
- ✅ Easy Docker deployment
**CONS:**
- ❌ Requires initial setup
- ❌ Less polished than Tailscale SaaS
- ❌ Self-managed (no cloud coordination)
---
### Tailscale (Official) - **SIMPLE ALTERNATIVE**
**Overview:**
Commercial mesh VPN service with generous free tier (up to 100 devices, 3 users).
**PROS:**
- ✅ Zero configuration
- ✅ Excellent reliability
- ✅ Free tier sufficient for home use
- ✅ Better NAT traversal out of the box
- ✅ Managed service
**CONS:**
- ❌ Relies on external service
- ❌ Privacy considerations (external control plane)
- ❌ Free tier limits
---
### Nebula
**Overview:**
Slack's open-source overlay network with built-in firewall capabilities.
**Key Differences:**
- Certificate-based authentication
- Built-in firewall (ACLs)
- Lighthouse coordination servers
- AES-256-GCM encryption
**Why Not Recommended:**
- More complex setup
- Smaller community than Tailscale/WireGuard
- Less polished tooling
- Steeper learning curve
---
### WireGuard
**Overview:**
Modern, lightweight VPN protocol built into Linux kernel.
**PROS:**
- ✅ Excellent performance (kernel-level)
- ✅ Simple protocol
- ✅ Widely supported
- ✅ Very secure
**CONS:**
- ❌ Point-to-point (not mesh)
- ❌ Manual configuration for mesh networking
- ❌ No built-in coordination
- ❌ More setup required for home use
---
## Comparison Matrix
| Solution | Existing OS | Web UI | Docker | GPU | NAS | Learning Curve | Storage | Best For |
|----------|------------|--------|--------|-----|-----|----------------|---------|----------|
| **Portainer + Docker** | ✅ YES | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐ | **EASY** | Minimal | **Best Overall** |
| **CasaOS** | ✅ YES | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | **EASIEST** | Minimal | Beginners |
| **Cockpit + Podman** | ✅ YES | ⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐ | Moderate | Minimal | Linux admins |
| **k3s/MicroK8s** | ✅ YES | ⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐ | **STEEP** | Low | Learning K8s |
| **TrueNAS Scale** | ❌ NO | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | Moderate | **HIGH** | NAS primary |
| **Unraid** | ❌ NO | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | Easy | Medium | Fresh install |
| **Proxmox VE** | ❌ NO | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐ | **STEEP** | Medium | Virtualization |
| **YunoHost** | ⚠️ Risky | ⭐⭐⭐⭐ | ⭐⭐ | ⭐ | ⭐⭐⭐ | Easy | Low | Not recommended |
---
## Final Recommendation: Portainer + Docker Compose
### Decision Factors
**Why Portainer Wins:**
1. ✅ **No OS Reinstall** - Works on existing Zorin OS
2. ✅ **Leverages Existing Docker** - Already have Docker 28.1.1 installed
3. ✅ **Minimal Storage Footprint** - Perfect for 481GB constraint
4. ✅ **Full RTX 2080 Ti Support** - Excellent for Jellyfin hardware transcoding
5. ✅ **Easy Learning Curve** - Rated 9.6/10 for ease of use
6. ✅ **Massive Ecosystem** - Thousands of pre-built containers
7. ✅ **Free and Open Source** - No licensing costs
8. ✅ **Quick Setup** - 15-30 minutes to get running
9. ✅ **Perfect for 16GB RAM / 481GB storage** - Minimal overhead
10. ✅ **Excellent Headscale Integration** - Simple Docker deployment
11. ✅ **Meets All Requirements** - Jellyfin, Nextcloud, file servers, web servers
12. ✅ **Active Community** - Extensive support and documentation
13. ✅ **Easy Extensibility** - Add services via web UI
14. ✅ **Web UI for Everything** - No command-line required for basic tasks
### When This Might Not Be Right
- If you need enterprise NAS features (ZFS snapshots, replication)
- If you want one-click app installation without any configuration (choose CasaOS)
- If you need advanced RAID configurations
- If you're planning major storage expansion (consider TrueNAS later)
### Alternative Consideration: CasaOS
**Choose CasaOS instead if:**
- You want the absolute easiest experience
- You prioritize beautiful UI over control
- You're completely new to self-hosting
- You want app-store simplicity
- You can sacrifice some control for ease-of-use
**Note:** You can also run both - CasaOS will recognize existing Docker containers managed by Portainer.
---
## Networking Recommendation
**Primary Choice: Headscale**
- Self-hosted Tailscale control server
- Full privacy and control
- Uses official Tailscale clients
- Docker container deployment
- No external dependencies
**Alternative: Tailscale Free Tier**
- Zero configuration
- Excellent reliability
- Free for personal use (100 devices, 3 users)
- Better NAT traversal out of the box
- Managed service (less maintenance)
**Recommendation:** Start with Headscale for full control, fall back to Tailscale if setup is too complex.
---
## Resource Links
### Portainer + Docker Compose
- Official Docs: https://docs.portainer.io/
- GPU Configuration: Search "Portainer GPU passthrough Docker Compose"
- Stack Templates: https://github.com/portainer/templates
### CasaOS
- Official Site: https://casaos.io/
- GitHub: https://github.com/IceWhaleTech/CasaOS
- Community: https://community.zimaspace.com/
### Headscale
- Official Docs: https://headscale.net/
- GitHub: https://github.com/juanfont/headscale
- Docker Setup: Check official documentation
### Jellyfin Hardware Transcoding
- Official Docs: https://jellyfin.org/docs/general/administration/hardware-acceleration/
- NVIDIA Guide: Jellyfin docs for NVIDIA-specific configuration
- RTX 2080 Ti: Fully supported, handles multiple 4K transcodes
### NVIDIA Container Toolkit
- Official Docs: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/
- Ubuntu Setup: Follow NVIDIA's Ubuntu installation guide
- Testing: Use nvidia-smi in containers to verify
### Docker Compose Examples
- Awesome Docker: https://github.com/veggiemonk/awesome-docker
- Compose Examples: https://github.com/docker/awesome-compose
- Media Server Stacks: Search GitHub for "jellyfin nextcloud docker-compose"
---
## Next Steps
Proceed to `implementation-plan.md` for detailed step-by-step implementation instructions with phases, tests, and validation checks.
---
*Research compiled from: TrueNAS community forums, Portainer documentation, CasaOS project, Jellyfin docs, NVIDIA Container Toolkit guides, Headscale documentation, Reddit homelab communities, and various technical blogs specializing in home server deployments (2024-2025)*
plans/completed/dashboard-strategy.md
-241
View File
@@ -1,241 +0,0 @@
# Unified Dashboard & External Access Strategy
> "One page to rule them all" - Unified interface for tower-of-joy services
> Created: 2025-11-11
## Overview
This document defines the strategy for creating a unified web interface that provides access to all tower-of-joy services through a single page with tabbed navigation.
## Solution: Organizr + Nginx Proxy Manager
**Organizr** provides the unified tabbed interface
**NPM** provides secure external access with SSL
## Architecture
```
Internet
↓
[DNS: home.schweitz.net]
↓
[Router: Port Forward 443 → 192.168.86.149:443]
↓
[Nginx Proxy Manager: 443]
↓
[Organizr: 9999] ←→ [Service Tabs via iframe]
├── Portainer (8001)
├── Uptime Kuma (3001)
├── Netdata (19999)
├── Heimdall (8888)
└── More services...
```
## URL Pattern: Single Domain Approach
**Recommended Pattern:**
```
https://home.schweitz.net → Organizr unified interface
```
**All services accessed through Organizr tabs:**
- Click "Portainer" tab → loads in iframe
- Click "Netdata" tab → loads in iframe
- Click "Uptime Kuma" tab → loads in iframe
**Why this pattern?**
- ✅ True "one page" experience
- ✅ Single SSL certificate
- ✅ Single URL to remember
- ✅ Centralized authentication
- ✅ Simple to maintain
## Alternative: Hybrid Subdomain Pattern
If some services need direct access (bypassing Organizr):
```
https://home.schweitz.net → Organizr (main interface)
https://portainer.home.schweitz.net → Direct Portainer access
https://netdata.home.schweitz.net → Direct Netdata access
```
**Requires:**
- Wildcard DNS: `*.home.schweitz.net → 192.168.86.149`
- Wildcard SSL cert OR individual certs per subdomain
## Service Configuration in Organizr
### Infrastructure Services (Primary Tabs)
| Service | Internal URL | Tab Name | Notes |
|---------|-------------|----------|-------|
| **Portainer** | http://192.168.86.149:8001 | Portainer | Container management |
| **Uptime Kuma** | http://192.168.86.149:3001 | Uptime | Service monitoring |
| **Netdata** | http://192.168.86.149:19999 | Metrics | System metrics |
| **Heimdall** | http://192.168.86.149:8888 | Dashboard | Alternative launcher |
### Optional Services (Additional Tabs)
| Service | Internal URL | Tab Name | Expose? |
|---------|-------------|----------|---------|
| **NPM Admin** | http://192.168.86.149:81 | NPM | Admin only - local access |
| **Headscale** | http://192.168.86.149:8085 | VPN | Admin only |
| **Ollama** | http://192.168.86.149:11434 | AI | API only, no UI |
### Future Application Services
| Service | Internal URL | Tab Name | Notes |
|---------|-------------|----------|-------|
| **Jellyfin** | http://192.168.86.149:8096 | Media | GPU transcoding |
| **Nextcloud** | http://192.168.86.149:8082 | Cloud | File storage |
## Iframe Embedding Challenges
### Known Issues
Some services block iframe embedding via `X-Frame-Options` header:
- **Netdata**: Can be configured to allow embedding
- **Portainer**: May require configuration
- **Uptime Kuma**: Generally works fine
### Solutions
**Option 1: Configure services to allow embedding**
Add to docker-compose environment:
```yaml
environment:
- X_FRAME_OPTIONS=SAMEORIGIN # Allow same-origin iframes
```
**Option 2: NPM header manipulation**
Configure NPM to strip/modify headers for internal access
**Option 3: Organizr "direct link" mode**
Services that don't work in iframes can open in new tab
## Security Layers
### Level 1: External Access (NPM)
- HTTPS with Let's Encrypt SSL
- External port 443 only
- DDoS protection via Cloudflare (optional)
### Level 2: Application Authentication (Organizr)
- User authentication in Organizr
- Role-based access control
- SSO integration (optional)
### Level 3: Service-Level Authentication
- Each service keeps its own auth
- Organizr can pass auth tokens (for supported services)
### Level 4: Network Security (Headscale)
- VPN access for sensitive admin tools
- Public: Jellyfin, Nextcloud
- Private (VPN only): Portainer, NPM, Netdata
## Implementation Steps
### Phase 1: Deploy Organizr
```bash
make deploy-organizr
```
### Phase 2: Configure Organizr
1. Access http://192.168.86.149:9999
2. Complete setup wizard
3. Create admin user
4. Add tabs for each service
### Phase 3: Configure NPM for External Access
1. Access NPM admin: http://192.168.86.149:81
2. Add proxy host:
- Domain: `home.schweitz.net`
- Forward to: `192.168.86.149:9999`
- Enable SSL with Let's Encrypt
- Force HTTPS redirect
### Phase 4: Configure Router Port Forwarding
```
External Port 443 → Internal 192.168.86.149:443 (NPM HTTPS)
External Port 80 → Internal 192.168.86.149:80 (NPM HTTP redirect)
```
### Phase 5: DNS Configuration
Point `home.schweitz.net` to your public IP
### Phase 6: Test & Secure
- Test external access: https://home.schweitz.net
- Verify SSL certificate
- Test all service tabs
- Configure Organizr authentication
- Review security settings
## Service Tab Recommendations
### Homepage Tab
- Quick status dashboard
- Links to most-used services
- System health indicators
### Essential Tabs (Always Visible)
- Portainer (container management)
- Uptime Kuma (monitoring)
- Netdata (metrics)
### Application Tabs (After deployment)
- Jellyfin (media)
- Nextcloud (files)
### Admin Tabs (Restricted)
- NPM (reverse proxy config)
- Headscale (VPN management)
## Maintenance
### Adding New Services
1. Deploy service via Portainer/Docker Compose
2. Add tab in Organizr settings
3. Test iframe embedding
4. Update this documentation
### SSL Certificate Renewal
- Automatic via Let's Encrypt (NPM handles this)
- Check NPM dashboard for expiry dates
### Security Updates
- Watchtower auto-updates containers (Phase 4)
- Review Organizr user access monthly
## Troubleshooting
### Service won't load in iframe
**Problem:** `X-Frame-Options` header blocking
**Solution:** Configure service to allow embedding, or use "open in new tab" mode
### External access not working
**Check:**
1. Router port forwarding configured (443 → 192.168.86.149:443)
2. DNS pointing to correct public IP
3. NPM proxy host configured correctly
4. SSL certificate generated successfully
### Authentication issues
**Check:**
1. Organizr user permissions
2. Service-specific authentication (each service has own login)
3. Consider implementing SSO for seamless experience
## Future Enhancements
### Potential Upgrades
- **Authelia**: Centralized authentication with 2FA
- **Cloudflare Tunnel**: Avoid port forwarding entirely
- **Custom Theme**: Brand Organizr to match preferences
- **API Integration**: Show live stats in Organizr homepage
---
**Next Steps:**
1. Deploy Organizr: `make deploy-organizr`
2. Configure tabs for existing services
3. Set up NPM proxy for external access
4. Test the unified interface
plans/completed/infrastructure-deployment-plan.md
-2464
View File
File diff suppressed because it is too large Load Diff
plans/completed/mesh-networking-strategy.md
-527
View File
@@ -1,527 +0,0 @@
# Mesh Network Access Strategy (Option B)
> Hybrid approach: Public access for media/files, VPN-only for admin tools
> All VPN access uses Headscale mesh IPs (10.99.0.x)
> Created: 2025-11-11
## Core Principles
**RULE: All external/public access MUST route through NPM proxy**
**Why this rule is mandatory:**
- ✅ **Let's Encrypt SSL**: Automatic certificate management in one place
- ✅ **Unified logging**: All external access logged in NPM
- ✅ **Security headers**: Consistent security policy (HSTS, CSP, etc.)
- ✅ **Access control**: Single point to manage public access
- ✅ **DDoS protection**: Can add Cloudflare/rate limiting at proxy level
- ✅ **No port sprawl**: Only ports 80/443 exposed externally
**Access Patterns:**
- **Internal/VPN access**: Direct mesh IPs → `http://10.99.0.1:8096`
- **External/Public access**: Through NPM → `https://media.schweitz.net` → NPM forwards to mesh IP
- **NEVER**: Direct port forwarding to services (except NPM and Headscale)
## Architecture Overview
```
┌─────────────────────────────────────────────────────────┐
│ Internet Users │
└──────────────────┬──────────────────┬───────────────────┘
│ │
┌──────────▼────────┐ ┌─────▼──────────────────┐
│ Public Access │ │ Headscale VPN │
│ (Port 443) │ │ (Port 8085) │
└──────────┬────────┘ └─────┬──────────────────┘
│ │
│ ┌──────▼──────────────────┐
│ │ VPN Mesh Network │
│ │ 10.99.0.0/16 │
│ │ │
│ │ tower-of-joy: 10.99.0.1│
│ │ laptop: 10.99.0.2 │
│ │ phone: 10.99.0.3 │
│ └──────┬──────────────────┘
│ │
┌─────────▼──────────────────▼─────────────────┐
│ tower-of-joy Services │
│ ┌────────────────────────────────────────┐ │
│ │ Public Services (via NPM) │ │
│ │ - Jellyfin (media) │ │
│ │ - Nextcloud (files) │ │
│ │ - Organizr (optional) │ │
│ └────────────────────────────────────────┘ │
│ ┌────────────────────────────────────────┐ │
│ │ VPN-Only Services (mesh IPs) │ │
│ │ - Portainer: 10.99.0.1:8001 │ │
│ │ - Netdata: 10.99.0.1:19999 │ │
│ │ - Uptime Kuma: 10.99.0.1:3001 │ │
│ │ - NPM Admin: 10.99.0.1:81 │ │
│ │ - Heimdall: 10.99.0.1:8888 │ │
│ └────────────────────────────────────────┘ │
└───────────────────────────────────────────────┘
```
## Service Access Matrix
| Service | Mesh IP Access | Public Access | Use Case |
|---------|---------------|---------------|----------|
| **Organizr** | ✅ http://10.99.0.1:9999 | ✅ https://home.schweitz.net | Unified dashboard |
| **Portainer** | ✅ http://10.99.0.1:8001 | ❌ VPN ONLY | Container management |
| **Netdata** | ✅ http://10.99.0.1:19999 | ❌ VPN ONLY | System metrics |
| **Uptime Kuma** | ✅ http://10.99.0.1:3001 | ❌ VPN ONLY | Service monitoring |
| **Heimdall** | ✅ http://10.99.0.1:8888 | ❌ VPN ONLY | Alternative dashboard |
| **NPM Admin** | ✅ http://10.99.0.1:81 | ❌ NEVER | Proxy config |
| **Headscale** | ✅ http://10.99.0.1:8085 | ✅ Public :8085 | VPN control plane |
| **Jellyfin** | ✅ http://10.99.0.1:8096 | ✅ https://media.schweitz.net | Media streaming |
| **Nextcloud** | ✅ http://10.99.0.1:8082 | ✅ https://cloud.schweitz.net | File storage |
| **Ollama** | ✅ http://10.99.0.1:11434 | ❌ VPN ONLY | ML API |
**Note:** Mesh IP `10.99.0.1` is assumed for tower-of-joy. Actual IP will be assigned by Headscale.
## Implementation Steps
### Phase 1: Connect tower-of-joy to Headscale
**First, get the server onto its own VPN mesh:**
```bash
# Install Tailscale client on tower-of-joy
curl -fsSL https://tailscale.com/install.sh | sh
# Connect to your Headscale server
sudo tailscale up --login-server=http://192.168.86.149:8085 \
--authkey=<your-preauth-key> \
--hostname=tower-of-joy
# Verify connection
tailscale status
# Should show: tower-of-joy with mesh IP (e.g., 10.99.0.1)
# Get the mesh IP assigned to tower-of-joy
tailscale ip -4
# Note this IP - you'll use it in Organizr configuration
```
**Verify from Headscale:**
```bash
# List all nodes in mesh
docker exec headscale headscale nodes list
# Should show:
# ID | Name | IP | Last Seen
# 1 | tower-of-joy | 10.99.0.1 | now
```
### Phase 2: Deploy Organizr
```bash
# Create config directory
mkdir -p ~/docker-data/organizr
# Deploy Organizr
docker compose -f stacks/organizr.yml up -d
# Verify running
docker ps | grep organizr
```
### Phase 3: Configure Organizr with Mesh IPs
**Access Organizr setup:**
- From local network: http://192.168.86.149:9999
- From VPN: http://10.99.0.1:9999
**Complete setup wizard:**
1. Choose installation type: "Personal"
2. Create admin user
3. Set timezone: Europe/Amsterdam
4. Complete setup
**Add tabs using mesh IPs:**
Navigate to: Settings → Tab Editor
#### Tab: Portainer
```
Tab Name: Portainer
Tab URL: http://10.99.0.1:8001
Tab Type: iframe
Category: Admin
Icon: docker
Enabled: Yes
Active: Yes
```
#### Tab: Netdata
```
Tab Name: Netdata
Tab URL: http://10.99.0.1:19999
Tab Type: iframe
Category: Monitoring
Icon: line-chart
Enabled: Yes
```
#### Tab: Uptime Kuma
```
Tab Name: Uptime
Tab URL: http://10.99.0.1:3001
Tab Type: iframe
Category: Monitoring
Icon: heartbeat
Enabled: Yes
```
#### Tab: Heimdall
```
Tab Name: Dashboard
Tab URL: http://10.99.0.1:8888
Tab Type: iframe
Category: Home
Icon: th
Enabled: Yes
```
#### Tab: Jellyfin (when deployed)
```
Tab Name: Media
Tab URL: http://10.99.0.1:8096
Tab Type: iframe
Category: Apps
Icon: film
Enabled: Yes
```
#### Tab: Nextcloud (when deployed)
```
Tab Name: Cloud
Tab URL: http://10.99.0.1:8082
Tab Type: iframe
Category: Apps
Icon: cloud
Enabled: Yes
```
### Phase 4: Configure NPM for Public Access
**Only expose these services publicly:**
Access NPM admin: http://10.99.0.1:81 (via VPN)
#### 1. Organizr (Public Dashboard)
```
Proxy Host Configuration:
Domain Names: home.schweitz.net
Scheme: http
Forward Hostname/IP: 10.99.0.1
Forward Port: 9999
✓ Block Common Exploits
✓ Websockets Support
SSL Tab:
✓ Force SSL
✓ HTTP/2 Support
✓ HSTS Enabled
Request New SSL Certificate (Let's Encrypt)
```
#### 2. Jellyfin (Public Media)
```
Proxy Host Configuration:
Domain Names: media.schweitz.net
Scheme: http
Forward Hostname/IP: 10.99.0.1
Forward Port: 8096
✓ Block Common Exploits
✓ Websockets Support
SSL Tab:
✓ Force SSL
✓ HTTP/2 Support
Request New SSL Certificate (Let's Encrypt)
```
#### 3. Nextcloud (Public Files)
```
Proxy Host Configuration:
Domain Names: cloud.schweitz.net
Scheme: http
Forward Hostname/IP: 10.99.0.1
Forward Port: 8082
✓ Block Common Exploits
✓ Websockets Support
SSL Tab:
✓ Force SSL
✓ HTTP/2 Support
Request New SSL Certificate (Let's Encrypt)
Custom Nginx Configuration:
client_max_body_size 10G; # Allow large file uploads
proxy_request_buffering off;
```
### Phase 5: DNS Configuration
**Required DNS records:**
```
home.schweitz.net A <your-public-ip>
media.schweitz.net A <your-public-ip>
cloud.schweitz.net A <your-public-ip>
```
**Or use wildcard:**
```
*.schweitz.net A <your-public-ip>
```
### Phase 6: Router Port Forwarding
**CRITICAL: ONLY these ports exposed to internet:**
```
External Port 443 → 192.168.86.149:443 (NPM HTTPS - ALL public services)
External Port 80 → 192.168.86.149:80 (NPM HTTP redirect to HTTPS)
External Port 8085 → 192.168.86.149:8085 (Headscale VPN control plane)
```
**⚠️ NEVER forward service ports directly!**
- ❌ DO NOT forward port 8096 (Jellyfin)
- ❌ DO NOT forward port 8082 (Nextcloud)
- ❌ DO NOT forward port 9999 (Organizr)
- ❌ DO NOT forward ANY service port except NPM and Headscale
**Why?**
- All public services MUST go through NPM for SSL and logging
- Direct port forwards bypass centralized security and logging
- NPM provides unified Let's Encrypt management
- NPM logs all external access for audit trails
## Access Patterns
### Scenario 1: Working from Home (Local Network)
**Can access via:**
- Local IPs: http://192.168.86.149:9999
- Mesh IPs: http://10.99.0.1:9999 (if VPN connected)
- Public domains: https://home.schweitz.net
**Best practice:** Use mesh IPs consistently for uniform experience
### Scenario 2: Remote Work (Connected to Headscale VPN)
**From laptop/phone on VPN:**
```bash
# Verify VPN connection
tailscale status
# Access Organizr
http://10.99.0.1:9999
# All tabs work with mesh IPs:
- Portainer: http://10.99.0.1:8001
- Netdata: http://10.99.0.1:19999
- Uptime Kuma: http://10.99.0.1:3001
```
**Accessing public services:**
- Can still use: https://media.schweitz.net (Jellyfin)
- Or direct mesh: http://10.99.0.1:8096
- Choose whichever is more convenient
### Scenario 3: Sharing with Family/Friends (No VPN)
**Public access only:**
- Jellyfin: https://media.schweitz.net
- Nextcloud: https://cloud.schweitz.net
- Organizr: https://home.schweitz.net (if you want public dashboard)
**Cannot access:**
- Admin tools (Portainer, Netdata, NPM) - VPN required
- They need Headscale VPN for admin access
## Security Configuration
### Organizr Authentication
**Enable auth for public access:**
Settings → User Management
- Create user accounts for family/friends
- Configure access levels:
- Admin: Full access to all tabs
- User: Only media/cloud tabs visible
- Guest: Read-only access
**Restrict admin tabs to admin users only:**
- Tab Editor → each admin tab → "Minimum Authentication" → Admin
### NPM Access Lists (Optional)
**For extra security on public services:**
Access Lists → Create "VPN Only"
```
Name: Headscale VPN Only
Allow: 10.99.0.0/16
Deny: all
```
Apply to sensitive proxy hosts if needed.
### Service-Level Authentication
**Each service maintains its own auth:**
- Portainer: Admin password
- Jellyfin: User accounts
- Nextcloud: User accounts
- Uptime Kuma: Admin password
**This is defense in depth:**
1. VPN layer (for admin tools)
2. Organizr layer (for organizing access)
3. Service layer (individual logins)
## Connecting Other Devices
### Laptop/Desktop
```bash
# Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh
# Connect to Headscale
sudo tailscale up --login-server=http://192.168.86.149:8085 \
--authkey=<your-preauth-key> \
--hostname=my-laptop
# Verify mesh access
curl http://10.99.0.1:9999
# Should load Organizr
```
### Phone (Android/iOS)
1. Install Tailscale app from store
2. In app settings:
- Use custom control server
- Server URL: http://<your-public-ip>:8085
- OR: http://192.168.86.149:8085 (if on local network)
3. Authenticate with pre-auth key
4. Open browser: http://10.99.0.1:9999
### Work Computer (Can't Install Software)
**Use public access only:**
- https://home.schweitz.net (Organizr - only non-admin tabs)
- https://media.schweitz.net (Jellyfin)
- https://cloud.schweitz.net (Nextcloud)
**Cannot access admin tools without VPN**
## Testing Checklist
### Phase 1: Local Access
- [ ] tower-of-joy connected to Headscale
- [ ] Mesh IP assigned (10.99.0.x)
- [ ] Can access services via mesh IP from tower-of-joy itself
### Phase 2: VPN Access from Another Device
- [ ] Connect laptop/phone to Headscale
- [ ] Verify mesh connectivity: `ping 10.99.0.1`
- [ ] Access Organizr: http://10.99.0.1:9999
- [ ] All tabs load correctly with mesh IPs
- [ ] Portainer accessible via mesh
- [ ] Netdata accessible via mesh
### Phase 3: Public Access
- [ ] DNS configured correctly
- [ ] NPM proxy hosts configured
- [ ] SSL certificates generated (green padlock)
- [ ] Access from public network (phone on mobile data):
- [ ] https://home.schweitz.net loads Organizr
- [ ] https://media.schweitz.net loads Jellyfin
- [ ] https://cloud.schweitz.net loads Nextcloud
- [ ] Admin tabs NOT accessible without VPN
### Phase 4: Security Validation
- [ ] Admin tools (Portainer, Netdata) not accessible from public internet
- [ ] Only exposed ports: 80, 443, 8085
- [ ] Organizr authentication working
- [ ] Service-level authentication working
## Advantages of This Architecture
### Mesh IP Benefits
✅ **Location independent:** Same IPs whether at home or remote
✅ **Secure by default:** Admin tools only via VPN
✅ **Simple routing:** No complex proxy rewrites
✅ **Flexible access:** Public and private services coexist
✅ **Future-proof:** Add devices easily, IPs don't change
✅ **No split-brain:** One set of URLs to remember
### NPM Proxy Benefits (For Public Access)
✅ **Centralized SSL:** All Let's Encrypt certs in one place
✅ **Unified logging:** All external access logged in NPM audit log
✅ **Security headers:** Consistent HSTS, CSP, X-Frame-Options
✅ **Access control:** Add rate limiting, IP blocking at proxy level
✅ **DDoS protection:** Can add Cloudflare in front of NPM
✅ **Port efficiency:** Only 2 ports exposed (80, 443)
### Compliance & Auditing
✅ **Audit trail:** NPM logs all external access attempts
✅ **SSL compliance:** Automatic certificate renewal
✅ **Security posture:** Single point to review/harden public access
✅ **Change management:** Proxy config changes tracked in one place
## Troubleshooting
### Can't connect to mesh IPs
**Check:**
```bash
# Verify Tailscale running
sudo systemctl status tailscaled
# Check mesh status
tailscale status
# Test connectivity
ping 10.99.0.1
```
### Organizr tabs not loading
**Issue:** Service blocking iframe embedding
**Solution:**
- Check browser console for errors
- Some services need `X-Frame-Options` configured
- Use "pseudo tab" mode (opens in new tab instead)
### Public access not working
**Check:**
1. DNS resolves to your public IP: `nslookup home.schweitz.net`
2. Router port forwarding configured
3. NPM proxy host using correct mesh IP (10.99.0.1)
4. SSL certificate valid
### Headscale connection fails
**Check:**
- Port 8085 accessible from internet
- Pre-auth key still valid
- Headscale service running: `docker logs headscale`
## Next Actions
1. **Connect tower-of-joy to Headscale** (get mesh IP)
2. **Deploy Organizr** (`make deploy-organizr`)
3. **Configure Organizr tabs** (using mesh IPs)
4. **Configure NPM** (public services only)
5. **Test VPN access** (from another device)
6. **Test public access** (from mobile data)
---
**This gives you the best of both worlds:**
- Secure admin access via VPN + mesh IPs
- Public access for media/files (family/friends)
- Single Organizr dashboard for everything
- No complex proxy rewrites
- Easy to add new devices
scripts/README.md
-53
View File
@@ -1,53 +0,0 @@
# Maintenance Scripts
This directory contains shell scripts for common maintenance tasks.
## Available Scripts
| Script | Description | Usage |
|--------|-------------|-------|
| `gpu-check.sh` | Verify GPU passthrough in containers | `./scripts/gpu-check.sh` |
| `health-check.sh` | Check all services and report status | `./scripts/health-check.sh` |
| `setup-kuma-monitors.sh` | Manual guide for configuring Uptime Kuma monitors | `./scripts/setup-kuma-monitors.sh` |
| `setup-kuma-monitors.py` | **Automated** Uptime Kuma monitor setup via API | `source .venv/bin/activate && python3 scripts/setup-kuma-monitors.py` |
| `backup-configs.sh` | Backup all Docker configs | `./scripts/backup-configs.sh` |
| `disk-usage.sh` | Report disk usage for SSD and HDD | `./scripts/disk-usage.sh` |
| `update-stacks.sh` | Pull latest images and update containers | `./scripts/update-stacks.sh <stack-name>` |
| `cleanup.sh` | Clean up unused Docker resources | `./scripts/cleanup.sh` |
## Making Scripts Executable
```bash
# Make all scripts executable
chmod +x scripts/*.sh
# Or individually
chmod +x scripts/health-check.sh
```
## Scheduling with Cron
Add to crontab for automated maintenance:
```bash
# Edit crontab
crontab -e
# Examples:
# Daily health check at 8 AM
0 8 * * * /home/jpmschweitzer/Projects/tower-of-joy/scripts/health-check.sh >> /var/log/tower-of-joy-health.log 2>&1
# Weekly cleanup on Sunday at 3 AM
0 3 * * 0 /home/jpmschweitzer/Projects/tower-of-joy/scripts/cleanup.sh
# Daily backup at 2 AM
0 2 * * * /home/jpmschweitzer/Projects/tower-of-joy/scripts/backup-configs.sh
```
## Script Guidelines
- All scripts should include error handling
- Use absolute paths for reliability
- Log output for debugging
- Exit with appropriate status codes (0 = success, non-zero = failure)
- Include help text with `-h` or `--help` flags
scripts/backup-configs.sh
-74
View File
@@ -1,74 +0,0 @@
#!/bin/bash
# Backup Docker Configurations Script
# Creates timestamped backup of all Docker configs
set -e
BACKUP_DIR="/mnt/media/backups/tower-of-joy-configs"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_PATH="${BACKUP_DIR}/backup_${TIMESTAMP}"
SOURCE_DIR="/home/jpmschweitzer/docker-data"
echo "=== Docker Configs Backup ==="
echo "Timestamp: $(date)"
echo ""
# Check if source exists
if [ ! -d "$SOURCE_DIR" ]; then
echo "❌ Source directory not found: $SOURCE_DIR"
exit 1
fi
# Create backup directory
echo "[1/4] Creating backup directory..."
mkdir -p "$BACKUP_PATH"
echo "✅ Created: $BACKUP_PATH"
echo ""
# Backup Docker configs
echo "[2/4] Backing up Docker configs..."
rsync -av --progress "$SOURCE_DIR/" "$BACKUP_PATH/" --exclude='cache' --exclude='*.log'
echo "✅ Configs backed up"
echo ""
# Backup stack files
echo "[3/4] Backing up stack definitions..."
mkdir -p "$BACKUP_PATH/stacks"
cp -r /home/jpmschweitzer/Projects/tower-of-joy/stacks/*.yml "$BACKUP_PATH/stacks/" 2>/dev/null || true
echo "✅ Stack files backed up"
echo ""
# Create backup manifest
echo "[4/4] Creating backup manifest..."
cat > "$BACKUP_PATH/MANIFEST.txt" << EOF
Backup created: $(date)
Hostname: $(hostname)
Docker version: $(docker --version)
Containers backed up:
$(docker ps --format ' - {{.Names}} ({{.Image}})')
Backup size: $(du -sh "$BACKUP_PATH" | cut -f1)
EOF
echo "✅ Manifest created"
echo ""
# Cleanup old backups (keep last 30 days)
echo "[Cleanup] Removing backups older than 30 days..."
find "$BACKUP_DIR" -maxdepth 1 -type d -name "backup_*" -mtime +30 -exec rm -rf {} \; 2>/dev/null || true
remaining=$(find "$BACKUP_DIR" -maxdepth 1 -type d -name "backup_*" | wc -l)
echo "✅ Kept $remaining recent backups"
echo ""
echo "=== Backup Complete ==="
echo "Location: $BACKUP_PATH"
echo "Size: $(du -sh "$BACKUP_PATH" | cut -f1)"
echo ""
# Verify backup
if [ -d "$BACKUP_PATH" ] && [ -f "$BACKUP_PATH/MANIFEST.txt" ]; then
echo "✅ Backup verification passed"
exit 0
else
echo "❌ Backup verification failed"
exit 1
fi
scripts/cleanup.sh
-72
View File
@@ -1,72 +0,0 @@
#!/bin/bash
# Docker Cleanup Script
# Safely removes unused containers, images, volumes, and networks
set -e
echo "=== Docker Cleanup ==="
echo "$(date)"
echo ""
# Show current usage
echo "[Current Docker Disk Usage]"
docker system df
echo ""
# Ask for confirmation
echo "This will remove:"
echo " - Stopped containers"
echo " - Unused networks"
echo " - Dangling images"
echo " - Build cache"
echo ""
read -p "Continue? (y/N) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Cleanup cancelled"
exit 0
fi
echo ""
echo "[1/4] Removing stopped containers..."
docker container prune -f
echo "✅ Done"
echo ""
echo "[2/4] Removing unused networks..."
docker network prune -f
echo "✅ Done"
echo ""
echo "[3/4] Removing dangling images..."
docker image prune -f
echo "✅ Done"
echo ""
echo "[4/4] Removing build cache..."
docker builder prune -f
echo "✅ Done"
echo ""
# Ask about unused images
echo ""
echo "[Optional] Remove ALL unused images (not just dangling)?"
echo "⚠️ This removes images not used by any container"
read -p "Remove unused images? (y/N) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
echo "Removing all unused images..."
docker image prune -a -f
echo "✅ Done"
fi
echo ""
echo "[New Docker Disk Usage]"
docker system df
echo ""
echo "=== Cleanup Complete ==="
echo ""
echo "Tip: Run 'docker volume prune' to remove unused volumes (use with caution!)"
scripts/disk-usage.sh
-65
View File
@@ -1,65 +0,0 @@
#!/bin/bash
# Disk Usage Report Script
# Shows detailed disk usage for SSD and HDD
echo "=== Disk Usage Report ==="
echo "$(date)"
echo ""
# Overall disk usage
echo "[Overall Disk Usage]"
df -h / /mnt/media 2>/dev/null || df -h /
echo ""
# SSD usage breakdown
echo "[SSD - Docker Configs] (/home/jpmschweitzer/docker-data)"
if [ -d "/home/jpmschweitzer/docker-data" ]; then
du -sh /home/jpmschweitzer/docker-data/* 2>/dev/null | sort -hr | head -10
else
echo "Directory not found"
fi
echo ""
# HDD usage breakdown
echo "[HDD - Media Content] (/mnt/media)"
if [ -d "/mnt/media" ]; then
du -sh /mnt/media/* 2>/dev/null | sort -hr
else
echo "⚠️ Media drive not mounted!"
fi
echo ""
# Docker system usage
echo "[Docker System Usage]"
docker system df
echo ""
# Largest containers
echo "[Largest Containers]"
docker ps --size --format "table {{.Names}}\t{{.Size}}" | head -11
echo ""
# Warn if getting full
ssd_percent=$(df /home/jpmschweitzer/docker-data 2>/dev/null | awk 'NR==2 {print $5}' | sed 's/%//')
hdd_percent=$(df /mnt/media 2>/dev/null | awk 'NR==2 {print $5}' | sed 's/%//')
echo "[Warnings]"
if [ -n "$ssd_percent" ] && [ "$ssd_percent" -gt 85 ]; then
echo "⚠️ SSD is ${ssd_percent}% full - consider cleanup!"
fi
if [ -n "$hdd_percent" ] && [ "$hdd_percent" -gt 85 ]; then
echo "⚠️ HDD is ${hdd_percent}% full - consider cleanup!"
fi
if [ -z "$hdd_percent" ]; then
echo "❌ Media drive not mounted at /mnt/media!"
fi
# Suggestions
echo ""
echo "[Cleanup Suggestions]"
echo "- Clean Docker: ./scripts/cleanup.sh"
echo "- Remove old images: docker image prune -a"
echo "- Check large files: du -sh /mnt/media/* | sort -hr"
echo ""
scripts/health-check.sh
-103
View File
@@ -1,103 +0,0 @@
#!/bin/bash
# Service Health Check Script
# Checks status of all critical services
set -e
echo "=== tower-of-joy Health Check ==="
echo "$(date)"
echo ""
# Define service check function
check_service() {
local name=$1
local port=$2
local container=$3
# Check container running
if docker ps --format '{{.Names}}' | grep -q "^${container}$"; then
# Check port responding
if curl -f -s -o /dev/null -w "%{http_code}" "http://localhost:${port}" > /dev/null 2>&1 || \
curl -f -s -o /dev/null "http://localhost:${port}" > /dev/null 2>&1; then
echo "✅ ${name} (port ${port})"
else
echo "⚠️ ${name} - container running but port ${port} not responding"
fi
else
echo "❌ ${name} - container not running"
fi
}
# Check infrastructure services
echo "[Infrastructure Services]"
check_service "Portainer" "8080" "portainer"
check_service "Nginx Proxy Manager" "8000" "nginx-proxy-manager"
check_service "Ollama" "11434" "ollama"
echo ""
# Check networking
echo "[Networking]"
check_service "Headscale" "8085" "headscale"
if command -v tailscale &> /dev/null; then
if tailscale status &> /dev/null; then
echo "✅ Tailscale connected"
else
echo "⚠️ Tailscale installed but not connected"
fi
else
echo "⚠️ Tailscale not installed"
fi
echo ""
# Check monitoring
echo "[Monitoring]"
check_service "Uptime Kuma" "3001" "uptime-kuma"
check_service "Netdata" "19999" "netdata"
check_service "Heimdall" "8888" "heimdall"
echo ""
# Check application services (if deployed)
echo "[Applications]"
check_service "Jellyfin" "8096" "jellyfin"
check_service "Nextcloud" "8082" "nextcloud"
check_service "Samba" "445" "samba"
echo ""
# Check storage
echo "[Storage]"
ssd_usage=$(df -h /home/jpmschweitzer/docker-data 2>/dev/null | awk 'NR==2 {print $5}' | sed 's/%//')
hdd_usage=$(df -h /mnt/media 2>/dev/null | awk 'NR==2 {print $5}' | sed 's/%//')
if [ -n "$ssd_usage" ]; then
if [ "$ssd_usage" -lt 80 ]; then
echo "✅ SSD: ${ssd_usage}% used"
elif [ "$ssd_usage" -lt 90 ]; then
echo "⚠️ SSD: ${ssd_usage}% used (getting full)"
else
echo "❌ SSD: ${ssd_usage}% used (critically full!)"
fi
else
echo "⚠️ SSD: Unable to check"
fi
if [ -n "$hdd_usage" ]; then
if [ "$hdd_usage" -lt 80 ]; then
echo "✅ HDD: ${hdd_usage}% used"
elif [ "$hdd_usage" -lt 90 ]; then
echo "⚠️ HDD: ${hdd_usage}% used (getting full)"
else
echo "❌ HDD: ${hdd_usage}% used (critically full!)"
fi
else
echo "❌ HDD: Not mounted at /mnt/media"
fi
echo ""
# Check Docker
echo "[Docker Status]"
running=$(docker ps -q | wc -l)
total=$(docker ps -aq | wc -l)
echo "Containers: ${running} running / ${total} total"
echo ""
echo "=== Health Check Complete ==="
scripts/update-stacks.sh
-82
View File
@@ -1,82 +0,0 @@
#!/bin/bash
# Update Docker Stacks Script
# Pull latest images and recreate containers
set -e
STACKS_DIR="/home/jpmschweitzer/Projects/tower-of-joy/stacks"
# Show usage
if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
echo "Usage: $0 [stack-name]"
echo ""
echo "Update a specific stack or all stacks"
echo ""
echo "Examples:"
echo " $0 portainer # Update only Portainer"
echo " $0 ollama # Update only Ollama"
echo " $0 # Update all stacks (interactive)"
echo ""
echo "Available stacks:"
ls -1 "$STACKS_DIR"/*.yml 2>/dev/null | xargs -n 1 basename | sed 's/.yml$//' | sed 's/^/ - /'
exit 0
fi
# Function to update a stack
update_stack() {
local stack_file=$1
local stack_name=$(basename "$stack_file" .yml)
echo ""
echo "=== Updating $stack_name ==="
# Pull latest images
echo "[1/3] Pulling latest images..."
docker compose -f "$stack_file" pull
# Recreate containers
echo "[2/3] Recreating containers..."
docker compose -f "$stack_file" up -d
# Verify containers running
echo "[3/3] Verifying containers..."
sleep 2
if docker compose -f "$stack_file" ps | grep -q "Up"; then
echo "✅ $stack_name updated successfully"
else
echo "⚠️ $stack_name may have issues, check logs"
fi
}
# If specific stack provided
if [ -n "$1" ]; then
STACK_FILE="$STACKS_DIR/$1.yml"
if [ -f "$STACK_FILE" ]; then
update_stack "$STACK_FILE"
else
echo "❌ Stack not found: $1"
echo "Available stacks:"
ls -1 "$STACKS_DIR"/*.yml 2>/dev/null | xargs -n 1 basename | sed 's/.yml$//' | sed 's/^/ - /'
exit 1
fi
else
# Interactive mode - update all stacks
echo "=== Update All Stacks ==="
echo ""
echo "This will update all deployed stacks to the latest images."
read -p "Continue? (y/N) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
for stack_file in "$STACKS_DIR"/*.yml; do
if [ -f "$stack_file" ]; then
update_stack "$stack_file"
fi
done
echo ""
echo "=== All Stacks Updated ==="
else
echo "Update cancelled"
exit 0
fi
fi
stacks/maintenance.yml
-69
View File
@@ -1,69 +0,0 @@
version: '3.8'
# Maintenance Runner - Scheduled Tasks & Scripts
# Purpose: Centralized container for running maintenance tasks (backups, cleanup, health checks, etc.)
# Ports: None (background service)
# GPU: No
# Storage: Read access to Docker data, write to backup location
services:
maintenance:
image: alpine:latest
container_name: maintenance
restart: unless-stopped
volumes:
# Docker configs to backup (read-only)
- /home/jpmschweitzer/docker-data:/data/docker-data:ro
# Host-based service configs to backup (read-only)
- /home/jpmschweitzer/.config/code-server:/data/code-server-config:ro
# Backup destination (read-write)
- /mnt/media/backups:/backups
# Maintenance scripts
- /home/jpmschweitzer/docker-data/maintenance/scripts:/scripts:ro
# Crontab configuration
- /home/jpmschweitzer/docker-data/maintenance/crontab:/etc/crontabs/root:ro
# Logs
- /home/jpmschweitzer/docker-data/maintenance/logs:/var/log/maintenance
# Docker socket for container management (read-only)
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
- TZ=Europe/Amsterdam
command: >
sh -c "
apk add --no-cache docker-cli &&
crond -f -l 2
"
# crond flags:
# -f: foreground (don't daemonize)
# -l 2: log level 2 (errors and info)
networks:
- docker-dataplane
networks:
docker-dataplane:
external: true
name: docker-dataplane
# Usage:
# 1. Create maintenance scripts in ~/docker-data/maintenance/scripts/
# 2. Define schedule in ~/docker-data/maintenance/crontab
# 3. Container will execute scripts per schedule
#
# Adding new maintenance tasks:
# 1. Create new script in scripts/ directory
# 2. Make it executable: chmod +x script-name.sh
# 3. Add to crontab with desired schedule
# 4. Restart container: docker restart maintenance
#
# View logs:
# - Cron logs: docker logs maintenance
# - Task logs: ~/docker-data/maintenance/logs/
#
# Manual task execution:
# docker exec maintenance /scripts/backup-configs.sh