restructure documentation
This commit is contained in:
@@ -41,7 +41,7 @@ This is the `tower-of-joy` project - a containerized home server infrastructure
|
||||
**Service Integration Policy:** A service deployment is INCOMPLETE until cross-service integrations are implemented. Every new service MUST be integrated with:
|
||||
- **Uptime Kuma:** Add health check monitor (use `scripts/setup-kuma-monitors.sh` as guide)
|
||||
- **Organizr:** Configure service in dashboard (Settings → Tab Editor, Homepage Items)
|
||||
- **CONTAINERS.md:** Document the service with full profile and configuration table
|
||||
- **docs/reference/CONTAINERS.md:** Document the service with full profile and configuration table
|
||||
|
||||
Services without monitoring and dashboard integration are considered unfinished and should not be marked as "complete" in STATUS.md or commit messages.
|
||||
|
||||
@@ -352,11 +352,11 @@ When deploying a NEW service, follow this complete checklist. A deployment is **
|
||||
- Test external access through proxy
|
||||
|
||||
**Phase 4: Documentation (MANDATORY)**
|
||||
- [ ] Add service profile to `CONTAINERS.md` with:
|
||||
- [ ] Add service profile to `docs/reference/CONTAINERS.md` with:
|
||||
- One-paragraph description
|
||||
- Complete configuration table
|
||||
- All dependencies listed
|
||||
- [ ] Add service to `CONTAINERS.md` quick reference tables:
|
||||
- [ ] Add service to `docs/reference/CONTAINERS.md` quick reference tables:
|
||||
- Service Access Matrix
|
||||
- Storage Distribution (if uses storage)
|
||||
- GPU-Enabled Services (if uses GPU)
|
||||
|
||||
+145
-8
@@ -1,6 +1,6 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to the tower-of-joy project will be documented in this file.
|
||||
All notable changes to the portainer-core project will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
@@ -8,12 +8,149 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
## [Unreleased]
|
||||
|
||||
### Planned
|
||||
- NVIDIA Container Toolkit installation
|
||||
- Portainer deployment
|
||||
- Nginx Proxy Manager deployment
|
||||
- Ollama ML infrastructure
|
||||
- Headscale secure networking
|
||||
- Monitoring stack (Uptime Kuma, Netdata, Heimdall)
|
||||
- AI Orchestrator Phase 2: Memory Systems (3-tier architecture with Qdrant)
|
||||
- AI Orchestrator Phases 3-6: Multi-agent workflows, tool integration, RAG, production hardening
|
||||
- Centralized database consolidation (PostgreSQL/MySQL container)
|
||||
- Disaster recovery and offsite backup strategy
|
||||
|
||||
## [0.7.1-gitea-deployment] - 2025-11-14
|
||||
|
||||
### Added
|
||||
- Gitea Git repository hosting service (port 3002, SSH port 2222)
|
||||
- PostgreSQL database backend for Gitea
|
||||
- NPM reverse proxy configuration for https://git.schweitz.net with Let's Encrypt SSL
|
||||
- Uptime Kuma monitoring integration for Gitea
|
||||
- Organizr dashboard integration for Gitea
|
||||
- Complete Gitea documentation in CONTAINERS.md
|
||||
|
||||
### Changed
|
||||
- Updated infrastructure status to reflect 19 deployed services
|
||||
|
||||
## [0.7.0-ai-orchestrator-phase1] - 2025-11-13
|
||||
|
||||
### Added
|
||||
- AI Orchestrator Phase 1: OpenAI-Compatible API
|
||||
- `/v1/chat/completions` endpoint with streaming and non-streaming support
|
||||
- `/v1/models` endpoint for model discovery
|
||||
- Model aliasing system (gpt-3.5-turbo → gemma:7b, etc.)
|
||||
- Ollama client with connection pooling
|
||||
- Pydantic request/response schemas
|
||||
- Server-Sent Events (SSE) streaming format
|
||||
- Comprehensive Phase 1 testing suite
|
||||
- 10/10 tests passing, zero issues
|
||||
- 245ms average response time
|
||||
- 100% OpenAI API compatibility verified
|
||||
- Phase 1 implementation guide and test results documentation
|
||||
|
||||
### Fixed
|
||||
- Model ID formatting issue (removed extra quotes in model names)
|
||||
|
||||
### Security
|
||||
- Deployed on isolated ai-dataplane network
|
||||
|
||||
## [0.6.0-applications] - 2025-11-13
|
||||
|
||||
### Added
|
||||
- Nextcloud cloud storage and collaboration platform (port 8082)
|
||||
- MariaDB database backend
|
||||
- Redis caching
|
||||
- NPM reverse proxy with https://cloud.schweitz.net
|
||||
- Database optimization (indices, bigint conversion)
|
||||
- Cron background jobs via maintenance container
|
||||
- Samba network file sharing (ports 139/445)
|
||||
- Media share (R/W)
|
||||
- Downloads share (R/W)
|
||||
- Backups share (R/O)
|
||||
- UFW firewall rules for Samba ports
|
||||
- Uptime Kuma multi-network bridge for monitoring all services
|
||||
|
||||
### Changed
|
||||
- Disabled host Samba service to prevent port conflicts
|
||||
- Relocated Nextcloud cron to maintenance container for centralized scheduling
|
||||
|
||||
### Fixed
|
||||
- Uptime Kuma network connectivity issues (added bridges to all service networks)
|
||||
|
||||
## [0.5.2-core-api] - 2025-11-13
|
||||
|
||||
### Added
|
||||
- Core API service for Open WebUI integration (port 8083)
|
||||
- Web scraper module with Trafilatura and BeautifulSoup
|
||||
- Infrastructure management API (Portainer/NPM/Kuma integration)
|
||||
- OpenAPI documentation at `/docs` endpoint
|
||||
- Health check endpoint
|
||||
- Uptime Kuma monitoring integration
|
||||
- Organizr dashboard integration
|
||||
|
||||
### Changed
|
||||
- Upgraded system Python from 3.8 (EOL) to 3.12
|
||||
|
||||
### Security
|
||||
- Runs as non-root user (uid 1000)
|
||||
- CORS configured for same-network access only
|
||||
|
||||
## [0.5.1-open-webui] - 2025-11-12
|
||||
|
||||
### Added
|
||||
- Open WebUI LLM chat interface (port 8081)
|
||||
- Built-in voice capabilities (local STT/TTS)
|
||||
- Ollama integration for local model inference
|
||||
- Uptime Kuma monitoring integration
|
||||
- Organizr dashboard integration (tab + homepage)
|
||||
- Complete Open WebUI documentation in CONTAINERS.md
|
||||
|
||||
## [0.5.0-optimization] - 2025-11-11
|
||||
|
||||
### Added
|
||||
- Phase 4: Optimization & Security
|
||||
- Watchtower for automatic container updates (daily at 4 AM)
|
||||
- Maintenance container for automated backups and scheduled tasks
|
||||
- Automated Docker config backups (daily at 3 AM, 30-day retention)
|
||||
- Docker log rotation configuration (10MB max, 3 files per container)
|
||||
- UFW firewall rules (SSH, Tailscale, infrastructure services)
|
||||
|
||||
### Security
|
||||
- Firewall enabled and configured for all public-facing services
|
||||
- Automated backup system with 30-day retention (~94MB per backup)
|
||||
|
||||
## [0.4.0-monitoring] - 2025-11-11
|
||||
|
||||
### Added
|
||||
- Phase 3: Monitoring Stack
|
||||
- Uptime Kuma service monitoring (port 3001)
|
||||
- Netdata real-time system metrics (port 19999)
|
||||
- Heimdall unified dashboard (port 8888)
|
||||
- Complete monitoring documentation
|
||||
|
||||
## [0.3.0-networking] - 2025-11-11
|
||||
|
||||
### Added
|
||||
- Phase 2: Networking & External Access
|
||||
- Headscale mesh VPN control server (port 8085)
|
||||
- Custom 10.99.0.0/16 network range
|
||||
- Homelab user and pre-auth key system
|
||||
- Device connection procedures for all platforms
|
||||
- Headscale setup documentation
|
||||
|
||||
## [0.2.0-foundation] - 2025-11-11
|
||||
|
||||
### Added
|
||||
- Phase 1: Foundation Setup
|
||||
- Portainer container management (port 8001, host networking)
|
||||
- Nginx Proxy Manager reverse proxy (port 81, host networking)
|
||||
- Ollama ML model serving (port 11434, GPU-enabled)
|
||||
- NVIDIA Container Toolkit (v1.17.9-1 for driver 470 compatibility)
|
||||
- GPU management via docker-compose deploy configuration
|
||||
- 4TB media drive mounted at /mnt/media
|
||||
- User added to docker group
|
||||
|
||||
### Fixed
|
||||
- Docker networking issues (iptables FORWARD chain, host networking solution)
|
||||
- AMP integration (kept on port 8080, no conflicts)
|
||||
|
||||
### Security
|
||||
- GPU passthrough configured securely
|
||||
- Storage permissions set for dual-disk strategy
|
||||
|
||||
## [0.1.0-planning] - 2025-11-11
|
||||
|
||||
@@ -99,4 +236,4 @@ Use these categories for changes:
|
||||
|
||||
---
|
||||
|
||||
*This changelog will be updated as phases are completed*
|
||||
*This changelog is updated as features are implemented and phases are completed*
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
# Implementation Plans
|
||||
|
||||
This document tracks all implementation plans across the portainer-core project.
|
||||
|
||||
## Active Plans
|
||||
|
||||
Current implementation work in progress:
|
||||
|
||||
### AI Orchestrator Enhancement
|
||||
**Location**: [plans/active/ai-orchestrator-plan.md](plans/active/ai-orchestrator-plan.md)
|
||||
**Status**: 🔄 Phase 2 in progress
|
||||
**Phases**:
|
||||
- ✅ Phase 1: OpenAI-Compatible API (Completed)
|
||||
- 🔄 Phase 2: Memory Systems (In Progress)
|
||||
- 📋 Phase 3: Multi-Model Management (Planned)
|
||||
- 📋 Phase 4: Reasoning & Chain-of-Thought (Planned)
|
||||
- 📋 Phase 5: Agentic Workflows (Planned)
|
||||
- 📋 Phase 6: Production Optimization (Planned)
|
||||
|
||||
### Memory Architecture
|
||||
**Location**: [plans/active/phase2-memory-architecture.md](plans/active/phase2-memory-architecture.md)
|
||||
**Status**: 🔄 In Progress
|
||||
**Description**: 3-tier memory system (ephemeral, short-term, long-term) for AI agents
|
||||
|
||||
### Security Implementation
|
||||
**Location**: [plans/active/security-implementation-plan.md](plans/active/security-implementation-plan.md)
|
||||
**Status**: 📋 Planning Phase
|
||||
**Description**: Google OAuth SSO via Authentik for external service access
|
||||
|
||||
---
|
||||
|
||||
## Completed Plans
|
||||
|
||||
Historical implementation plans that have been finished:
|
||||
|
||||
### Infrastructure Deployment (Phases 1-4)
|
||||
**Location**: [plans/completed/infrastructure-deployment-plan.md](plans/completed/infrastructure-deployment-plan.md)
|
||||
**Completed**: November 2025
|
||||
**Phases**:
|
||||
- ✅ Phase 1: Foundation (Portainer, NPM, Ollama)
|
||||
- ✅ Phase 2: Networking (Headscale mesh VPN)
|
||||
- ✅ Phase 3: Monitoring (Uptime Kuma, Netdata, Heimdall)
|
||||
- ✅ Phase 4: Optimization (Watchtower, Duplicati)
|
||||
|
||||
### AI Orchestrator Phase 1
|
||||
**Location**: [plans/completed/ai-orchestrator-phase1-guide.md](plans/completed/ai-orchestrator-phase1-guide.md)
|
||||
**Completed**: November 2025
|
||||
**Deliverables**: OpenAI-compatible API with model routing, streaming, function calling
|
||||
|
||||
### AI Orchestrator Phase 1 Testing
|
||||
**Location**: [plans/completed/ai-orchestrator-phase1-tests.md](plans/completed/ai-orchestrator-phase1-tests.md)
|
||||
**Results**: 10/10 tests passed, zero issues found
|
||||
|
||||
### Architecture Research
|
||||
**Location**: [plans/completed/architecture-research.md](plans/completed/architecture-research.md)
|
||||
**Completed**: October 2025
|
||||
**Decision**: Portainer + Docker Compose for container orchestration
|
||||
|
||||
### Mesh Networking Strategy
|
||||
**Location**: [plans/completed/mesh-networking-strategy.md](plans/completed/mesh-networking-strategy.md)
|
||||
**Completed**: November 2025
|
||||
**Solution**: Headscale (self-hosted Tailscale) for secure mesh VPN
|
||||
|
||||
### Dashboard Consolidation Strategy
|
||||
**Location**: [plans/completed/dashboard-strategy.md](plans/completed/dashboard-strategy.md)
|
||||
**Completed**: November 2025
|
||||
**Solution**: Organizr with custom service control widgets
|
||||
|
||||
---
|
||||
|
||||
## Plan Management
|
||||
|
||||
### Creating New Plans
|
||||
|
||||
1. Create plan in `plans/active/` directory
|
||||
2. Add entry to "Active Plans" section above
|
||||
3. Update STATUS.md with phase tracking
|
||||
4. Link from relevant documentation
|
||||
|
||||
### Completing Plans
|
||||
|
||||
1. Mark all phases as ✅ in the plan document
|
||||
2. Move from `plans/active/` to `plans/completed/`
|
||||
3. Update this file (move to "Completed Plans" section)
|
||||
4. Update STATUS.md
|
||||
5. Update CHANGELOG.md with release notes
|
||||
|
||||
### Plan Template
|
||||
|
||||
```markdown
|
||||
# [Feature Name] Implementation Plan
|
||||
|
||||
## Overview
|
||||
Brief description of the feature/improvement.
|
||||
|
||||
## Motivation
|
||||
Why this change is needed.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: [Name]
|
||||
**Status**: 📋 Planned / 🔄 In Progress / ✅ Completed
|
||||
**Duration**: Estimated effort
|
||||
**Deliverables**:
|
||||
- [ ] Task 1
|
||||
- [ ] Task 2
|
||||
|
||||
## Success Criteria
|
||||
How to determine if implementation is complete.
|
||||
|
||||
## Testing Strategy
|
||||
How the feature will be validated.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Links
|
||||
|
||||
- [Project Status](STATUS.md) - Current phase and progress tracking
|
||||
- [Documentation Index](README.md) - All project documentation
|
||||
- [Active Plans](plans/active/) - Current implementation work
|
||||
- [Completed Plans](plans/completed/) - Historical implementations
|
||||
@@ -1,35 +1,58 @@
|
||||
# tower-of-joy
|
||||
# portainer-core
|
||||
|
||||
> Self-hosted home server infrastructure with GPU-accelerated ML model serving, media streaming, and secure remote access
|
||||
> Self-hosted home server infrastructure with GPU-accelerated ML, AI orchestration, media streaming, and secure remote access
|
||||
|
||||
## Overview
|
||||
**Main Dashboard:** https://home.schweitz.net (Organizr)
|
||||
|
||||
**tower-of-joy** is a containerized home server platform running on the "tower-of-joy" system, leveraging Portainer + Docker Compose for service orchestration. The infrastructure supports GPU-accelerated workloads (ML inference via Ollama, media transcoding via Jellyfin) while maintaining a clean separation between performance-critical configs (SSD) and bulk content storage (HDD).
|
||||
## Quick Links
|
||||
|
||||
## Quick Start
|
||||
### Getting Started
|
||||
- [System Specifications](docs/reference/SYSTEM.md) - Hardware and software details
|
||||
- [Container Reference](docs/reference/CONTAINERS.md) - All deployed services
|
||||
- [Current Status](STATUS.md) - Implementation progress and phase tracking
|
||||
|
||||
**Main Dashboard:** https://home.schweitz.net (Organizr - unified interface for all services)
|
||||
### Implementation Plans
|
||||
- [Implementation Plans](PLANS.md) - Master plan tracker
|
||||
- [Active Plans](plans/active/) - Current development work
|
||||
- [Completed Plans](plans/completed/) - Historical implementations
|
||||
|
||||
### Documentation Index
|
||||
|
||||
## System Specifications
|
||||
#### Architecture & Design
|
||||
- [Shared Infrastructure Architecture](docs/architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md) - PostgreSQL/Redis shared infrastructure
|
||||
|
||||
- **Host:** tower-of-joy (Zorin OS 16.3 / Ubuntu 20.04)
|
||||
- **CPU:** Intel i7-6700 (4C/8T @ 3.40GHz)
|
||||
- **RAM:** 16GB
|
||||
- **GPU:** NVIDIA RTX 2080 Ti (11GB VRAM)
|
||||
- **Storage:**
|
||||
- **SSD (489GB):** Configs, databases, Docker images → `/home/jpmschweitzer/docker-data/`
|
||||
- **HDD (3.7TB):** Media, user content, backups → `/mnt/media/`
|
||||
#### Operational Guides
|
||||
- [Backup Procedures](docs/guides/backup-procedures.md) - Backup strategies and procedures
|
||||
- [Code-Server Setup](docs/guides/code-server-setup.md) - Browser-based IDE configuration
|
||||
- [Connect Devices Guide](docs/guides/connect-devices-guide.md) - Headscale VPN setup
|
||||
- [GPU Docker Configuration](docs/guides/gpu-docker-config.md) - NVIDIA GPU passthrough
|
||||
- [Headscale Setup](docs/guides/headscale-setup.md) - Mesh VPN deployment
|
||||
- [NPM Logging Guide](docs/guides/npm-logging-guide.md) - Nginx Proxy Manager logging
|
||||
|
||||
## Architecture
|
||||
#### Services
|
||||
- [Core API](docs/services/core-api.md) - Infrastructure management and AI orchestration
|
||||
- [Organizr Widgets](docs/services/organizr-widgets.md) - Service control dashboard
|
||||
|
||||
#### Reference
|
||||
- [Stacks Reference](docs/reference/stacks.md) - All Docker Compose stacks
|
||||
- [Scripts Reference](docs/reference/scripts.md) - Maintenance automation
|
||||
- [Automation Reference](docs/reference/AUTOMATION.md) - Portainer REST API usage
|
||||
- [Container Reference](docs/reference/CONTAINERS.md) - Complete container profiles
|
||||
- [System Reference](docs/reference/SYSTEM.md) - Hardware specifications
|
||||
- [Changelog](CHANGELOG.md) - Version history
|
||||
|
||||
### For AI Agents
|
||||
- [Agent Guidelines](AGENTS.md) - **REQUIRED READING** for all LLM coding agents
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Infrastructure Layer │
|
||||
│ ├── Portainer (8001) - Container mgmt │
|
||||
│ ├── Portainer (8080) - Container mgmt │
|
||||
│ ├── PostgreSQL Shared (5432) - DB │
|
||||
│ ├── Redis Shared (6379) - Cache │
|
||||
│ ├── NPM (81) - Reverse proxy │
|
||||
│ ├── NPM (8000) - Reverse proxy │
|
||||
│ └── Ollama (11434) - ML models [GPU] │
|
||||
├─────────────────────────────────────────┤
|
||||
│ Networking Layer │
|
||||
@@ -39,14 +62,14 @@
|
||||
│ Monitoring Layer │
|
||||
│ ├── Uptime Kuma (3001) - Uptime │
|
||||
│ ├── Netdata (19999) - Metrics │
|
||||
│ └── Organizr (9999) - Dashboard │
|
||||
│ └── Organizr (8084) - Dashboard │
|
||||
├─────────────────────────────────────────┤
|
||||
│ Optimization Layer │
|
||||
│ ├── Watchtower - Auto-updates │
|
||||
│ └── Maintenance - Automated backups │
|
||||
├─────────────────────────────────────────┤
|
||||
│ Application Layer │
|
||||
│ ├── Open WebUI (82) - LLM chat UI │
|
||||
│ ├── Open WebUI (8081) - LLM chat UI │
|
||||
│ ├── Core API (8083) - Infra mgmt │
|
||||
│ ├── Jellyfin (8096) - Media [GPU] │
|
||||
│ ├── Nextcloud (8082) - Cloud storage │
|
||||
@@ -58,59 +81,28 @@
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
tower-of-joy/
|
||||
├── stacks/ # Docker Compose files (version-controlled)
|
||||
│ ├── portainer.yml
|
||||
│ ├── nginx-proxy-manager.yml
|
||||
│ ├── ollama.yml
|
||||
│ ├── headscale.yml
|
||||
│ ├── jellyfin.yml
|
||||
│ ├── nextcloud.yml
|
||||
portainer-core/
|
||||
├── plans/ # Implementation plans
|
||||
│ ├── active/ # Current development work
|
||||
│ └── completed/ # Historical implementations
|
||||
├── docs/ # Documentation
|
||||
│ ├── architecture/ # Design documents
|
||||
│ ├── guides/ # Setup and operational guides
|
||||
│ ├── services/ # Service-specific documentation
|
||||
│ └── reference/ # Quick reference materials
|
||||
├── stacks/ # Docker Compose files (version-controlled)
|
||||
├── scripts/ # Maintenance automation
|
||||
├── services/ # Service source code
|
||||
│ ├── core-api/ # Infrastructure management API
|
||||
│ └── ...
|
||||
├── scripts/ # Maintenance automation
|
||||
│ ├── health-check.sh
|
||||
│ ├── gpu-check.sh
|
||||
│ ├── backup-configs.sh
|
||||
│ ├── disk-usage.sh
|
||||
│ └── cleanup.sh
|
||||
├── containers/ # Research & implementation docs
|
||||
│ ├── research.md
|
||||
│ └── implementation-plan.md
|
||||
├── Makefile # Common operations
|
||||
├── STATUS.md # Current phase tracking
|
||||
├── CHANGELOG.md # Version history
|
||||
├── AGENTS.md # AI agent guidelines
|
||||
└── SYSTEM.md # Hardware documentation
|
||||
├── organizr-widgets/ # Dashboard widgets
|
||||
├── AGENTS.md # AI agent guidelines (single source of truth)
|
||||
├── README.md # This file (documentation index)
|
||||
├── PLANS.md # Implementation plan tracker
|
||||
├── STATUS.md # Current phase tracking
|
||||
└── CHANGELOG.md # Version history
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- **[CONTAINERS.md](CONTAINERS.md)** - Complete container reference guide with specs and access details
|
||||
- **[docs/SHARED_INFRASTRUCTURE_ARCHITECTURE.md](docs/SHARED_INFRASTRUCTURE_ARCHITECTURE.md)** - PostgreSQL/Redis shared infrastructure design
|
||||
- **[AGENTS.md](AGENTS.md)** - Guidelines for AI coding agents (conventions, testing, commits)
|
||||
- **[STATUS.md](STATUS.md)** - Current implementation phase and progress
|
||||
- **[CHANGELOG.md](CHANGELOG.md)** - Version history and completed work
|
||||
- **[SYSTEM.md](SYSTEM.md)** - Detailed hardware and software specs
|
||||
|
||||
## Development Setup
|
||||
|
||||
### Python Environment
|
||||
|
||||
Some automation scripts require Python dependencies. A virtual environment is provided:
|
||||
|
||||
```bash
|
||||
# Activate virtual environment
|
||||
source .venv/bin/activate
|
||||
|
||||
# Install/update dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Deactivate when done
|
||||
deactivate
|
||||
```
|
||||
|
||||
**Note:** The `.venv/` directory is gitignored and must be created on each system.
|
||||
|
||||
## Common Commands
|
||||
|
||||
### Infrastructure Management
|
||||
@@ -132,94 +124,96 @@ make update-jellyfin # Update Jellyfin to latest
|
||||
make stop-nextcloud # Stop Nextcloud stack
|
||||
```
|
||||
|
||||
### Phase Deployment
|
||||
See [Stacks Reference](docs/reference/stacks.md) for complete stack inventory and deployment procedures.
|
||||
|
||||
## Development Setup
|
||||
|
||||
### Python Environment
|
||||
|
||||
Some automation scripts require Python dependencies:
|
||||
|
||||
```bash
|
||||
make deploy-phase1 # Deploy foundation (Portainer, NPM, Ollama)
|
||||
make deploy-phase2 # Deploy networking (Headscale)
|
||||
make deploy-phase3 # Deploy monitoring (Uptime Kuma, Netdata, Heimdall)
|
||||
make deploy-phase4 # Deploy optimization (Watchtower, Maintenance)
|
||||
make deploy-apps # Deploy applications (Jellyfin, Nextcloud, Samba)
|
||||
# Activate virtual environment
|
||||
source .venv/bin/activate
|
||||
|
||||
# Install/update dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Deactivate when done
|
||||
deactivate
|
||||
```
|
||||
|
||||
## Service Ports
|
||||
### Service Development
|
||||
|
||||
See individual service documentation:
|
||||
- [Core API Development](docs/services/core-api.md#development)
|
||||
|
||||
## Service Ports Reference
|
||||
|
||||
| Service | Port | Description |
|
||||
|---------|------|-------------|
|
||||
| **Portainer** | 8001 | Container management UI |
|
||||
| **PostgreSQL Shared** | 5432 | Shared database server (internal) |
|
||||
| **Redis Shared** | 6379 | Shared cache server (internal) |
|
||||
| **Nginx Proxy Manager** | 81 | Reverse proxy admin |
|
||||
| **Open WebUI** | 82 | LLM chat interface |
|
||||
| **Ollama** | 11434 | ML model API |
|
||||
| **Portainer** | 8080 | Container management UI |
|
||||
| **Nginx Proxy Manager** | 8000 | Reverse proxy admin |
|
||||
| **Open WebUI** | 8081 | LLM chat interface |
|
||||
| **Nextcloud** | 8082 | Cloud storage |
|
||||
| **Core API** | 8083 | Infrastructure management API |
|
||||
| **Code-Server** | 8084 | Browser-based IDE (localhost only) |
|
||||
| **Organizr** | 8084 | Unified dashboard |
|
||||
| **Headscale** | 8085 | VPN control server |
|
||||
| **Jellyfin** | 8096 | Media streaming |
|
||||
| **Nextcloud** | 8082 | Cloud storage |
|
||||
| **Uptime Kuma** | 3001 | Service monitoring |
|
||||
| **Gitea** | 3002 | Git repository hosting |
|
||||
| **Gitea SSH** | 2222 | Git SSH access |
|
||||
| **PostgreSQL Shared** | 5432 | Shared database (internal) |
|
||||
| **Redis Shared** | 6379 | Shared cache (internal) |
|
||||
| **Qdrant** | 6333, 6334 | Vector database |
|
||||
| **Ollama** | 11434 | ML model API |
|
||||
| **Netdata** | 19999 | System monitoring |
|
||||
| **Organizr** | 9999 | Unified dashboard |
|
||||
|
||||
See [Stacks Reference](docs/reference/stacks.md#port-allocation) for complete port allocation.
|
||||
|
||||
## GPU Services
|
||||
|
||||
Two services leverage the RTX 2080 Ti for GPU acceleration:
|
||||
Two services leverage the RTX 2080 Ti:
|
||||
|
||||
1. **Ollama** (ML inference)
|
||||
- Supports 3B-13B parameter models
|
||||
- Recommended: llama3.2:3b, mistral:7b, codellama:7b
|
||||
1. **Ollama** - ML model inference (3B-13B parameter models)
|
||||
2. **Jellyfin** - Hardware video transcoding (NVENC)
|
||||
|
||||
2. **Jellyfin** (Media transcoding)
|
||||
- NVIDIA NVENC hardware encoding
|
||||
- Can handle multiple 4K transcodes simultaneously
|
||||
See [GPU Docker Configuration](docs/guides/gpu-docker-config.md) for setup.
|
||||
|
||||
## Storage Strategy
|
||||
|
||||
**SSD (Performance-Critical):**
|
||||
- Docker configs
|
||||
- Application databases
|
||||
- Cache directories
|
||||
- Container images
|
||||
**SSD (Performance):** `/home/jpmschweitzer/docker-data/`
|
||||
- Docker configs, databases, cache, container images
|
||||
|
||||
**HDD (Capacity-Critical):**
|
||||
- Media files (Jellyfin)
|
||||
- User data (Nextcloud)
|
||||
- Game server worlds (AMP)
|
||||
- Backups
|
||||
**HDD (Capacity):** `/mnt/media/`
|
||||
- Media files, user data, backups
|
||||
|
||||
## Current Status
|
||||
See [Stacks Reference](docs/reference/stacks.md#storage-convention) for details.
|
||||
|
||||
**Phase:** Planning & Documentation Complete ✅
|
||||
## Current Phase
|
||||
|
||||
**Next Steps:**
|
||||
1. Review implementation plan
|
||||
2. Verify prerequisites (Docker, GPU, disk space)
|
||||
3. Begin Phase 1: Foundation Setup
|
||||
**Phase 2** of AI Orchestrator Enhancement (Memory Systems) 🔄 In Progress
|
||||
|
||||
See [STATUS.md](STATUS.md) for detailed progress tracking.
|
||||
|
||||
## Contributing
|
||||
|
||||
This is a personal infrastructure project. For AI agents working on this codebase:
|
||||
- Read [AGENTS.md](AGENTS.md) for guidelines
|
||||
This is a personal infrastructure project. For AI agents:
|
||||
- **Read [AGENTS.md](AGENTS.md) first** - Mandatory guidelines
|
||||
- Follow conventional commit format
|
||||
- Test GPU access before deploying GPU services
|
||||
- Update STATUS.md when completing phases
|
||||
|
||||
## License
|
||||
|
||||
Personal infrastructure project - not licensed for reuse.
|
||||
|
||||
## Resources
|
||||
|
||||
- **Portainer:** https://docs.portainer.io/
|
||||
- **Ollama:** https://github.com/ollama/ollama
|
||||
- **Headscale:** https://headscale.net/
|
||||
- **Jellyfin:** https://jellyfin.org/docs/
|
||||
- **Nextcloud:** https://docs.nextcloud.com/
|
||||
- [Portainer Documentation](https://docs.portainer.io/)
|
||||
- [Ollama](https://github.com/ollama/ollama)
|
||||
- [Headscale](https://headscale.net/)
|
||||
- [Jellyfin](https://jellyfin.org/docs/)
|
||||
- [Nextcloud](https://docs.nextcloud.com/)
|
||||
|
||||
---
|
||||
|
||||
**Version:** 0.5.0-optimization
|
||||
**Last Updated:** 2025-11-16
|
||||
**Version:** 0.7.1
|
||||
**Last Updated:** 2025-11-20
|
||||
**System:** tower-of-joy
|
||||
|
||||
@@ -1,148 +1,40 @@
|
||||
# tower-of-joy Project Status
|
||||
# Project Status
|
||||
|
||||
> Last Updated: 2025-11-14
|
||||
> Version: 0.7.1-gitea-deployment
|
||||
> **Last Updated:** 2025-11-20
|
||||
> **Version:** 0.7.1-gitea-deployment
|
||||
|
||||
## Current Phase
|
||||
|
||||
**Phase:** AI Orchestrator Development - Phase 1 (Foundation)
|
||||
**Status:** ✅ **COMPLETED**
|
||||
**Started:** 2025-11-13
|
||||
**Completed:** 2025-11-13
|
||||
|
||||
**Next Phase:** AI Orchestrator - Phase 2 (Memory Systems)
|
||||
**Active Work:** AI Orchestrator - Phase 2 (Memory Systems)
|
||||
**Status:** 🔄 **IN PROGRESS**
|
||||
|
||||
## Implementation Progress
|
||||
See [PLANS.md](PLANS.md) for complete implementation roadmap and [CHANGELOG.md](CHANGELOG.md) for version history.
|
||||
|
||||
### ✅ Completed
|
||||
## In Progress
|
||||
|
||||
**Planning & Documentation:**
|
||||
- [x] Research home server solutions (see containers/research.md)
|
||||
- [x] Architecture decision: Portainer + Docker Compose
|
||||
- [x] Implementation plan created (see containers/implementation-plan.md)
|
||||
- [x] AGENTS.md populated with project-specific guidelines
|
||||
- [x] Storage strategy defined (SSD for configs, HDD for content)
|
||||
|
||||
**Phase 1: Foundation Setup:**
|
||||
- [x] Install NVIDIA Container Toolkit (v1.17.9-1 - downgraded for driver 470 compatibility)
|
||||
- [x] Deploy Portainer (port 8001, host networking)
|
||||
- [x] Configure GPU management (via docker-compose deploy configuration)
|
||||
- [x] Mount 4TB media drive at /mnt/media
|
||||
- [x] Configure AMP integration (kept on port 8080, no conflicts)
|
||||
- [x] Add user to docker group
|
||||
- [x] Deploy Nginx Proxy Manager (port 81, host networking)
|
||||
- [x] Deploy Ollama ML infrastructure (port 11434, GPU-enabled)
|
||||
- [x] Fix Docker networking issues (iptables FORWARD chain, host networking solution)
|
||||
|
||||
**Phase 2: Networking & External Access:**
|
||||
- [x] Deploy Headscale (port 8085, mesh VPN control server)
|
||||
- [x] Configure Headscale with custom 10.99.0.0/16 network range
|
||||
- [x] Create homelab user and generate pre-auth keys
|
||||
- [x] Document connection procedures for all device types
|
||||
|
||||
**Phase 3: Monitoring & Management:**
|
||||
- [x] Deploy Uptime Kuma (port 3001, service uptime monitoring)
|
||||
- [x] Deploy Netdata (port 19999, real-time system metrics)
|
||||
- [x] Deploy Heimdall dashboard (port 8888, unified dashboard)
|
||||
|
||||
**Phase 4: Optimization & Security:**
|
||||
- [x] Deploy Watchtower (automatic container updates, daily at 4 AM)
|
||||
- [x] Configure Docker log rotation (10MB max, 3 files per container)
|
||||
- [x] Configure UFW firewall (SSH, Tailscale, infrastructure services allowed)
|
||||
- [x] Deploy maintenance container (scheduled backups & future maintenance tasks)
|
||||
- [x] Configure automated Docker config backups (daily at 3 AM, 30-day retention, ~94MB/backup)
|
||||
|
||||
**Application Deployment:**
|
||||
- [x] Deploy Open WebUI (LLM chat interface, port 82)
|
||||
- [x] Integrate Open WebUI with Uptime Kuma (health monitoring)
|
||||
- [x] Integrate Open WebUI with Organizr (dashboard tab + homepage)
|
||||
- [x] Document Open WebUI in CONTAINERS.md
|
||||
- [x] Deploy Core API (OpenAPI functions for Open WebUI, port 8083)
|
||||
- [x] Integrate Core API with Uptime Kuma (health monitoring)
|
||||
- [x] Integrate Core API with Organizr (dashboard tab)
|
||||
- [x] Document Core API in CONTAINERS.md
|
||||
- [x] Upgrade system to Python 3.12 (from EOL 3.8)
|
||||
|
||||
**AI Orchestrator Development (Phase 1):**
|
||||
- [x] Implement OpenAI-compatible `/v1/chat/completions` endpoint
|
||||
- [x] Implement OpenAI-compatible `/v1/models` endpoint
|
||||
- [x] Add streaming support (Server-Sent Events format)
|
||||
- [x] Add non-streaming response mode
|
||||
- [x] Implement model aliasing system (gpt-3.5-turbo → gemma:7b, etc.)
|
||||
- [x] Create Ollama client with connection pooling
|
||||
- [x] Add proper request/response schemas (Pydantic models)
|
||||
- [x] Deploy to ai-dataplane network with hot-reload
|
||||
- [x] Test streaming and non-streaming responses
|
||||
- [x] Update CONTAINERS.md documentation
|
||||
- [x] Deploy Nextcloud (cloud storage and collaboration platform, port 8082)
|
||||
- [x] Configure Nextcloud with MariaDB, Redis, and reverse proxy (https://cloud.schweitz.net)
|
||||
- [x] Optimize Nextcloud (database indices, bigint conversion, cron background jobs)
|
||||
- [x] Relocate Nextcloud cron to maintenance container
|
||||
- [x] Integrate Nextcloud with Uptime Kuma (HTTP monitoring)
|
||||
- [x] Integrate Nextcloud with Organizr (dashboard tab)
|
||||
- [x] Document Nextcloud in CONTAINERS.md
|
||||
- [x] Deploy Samba (network file sharing, ports 139/445)
|
||||
- [x] Configure Samba shares (Media R/W, Downloads R/W, Backups R/O)
|
||||
- [x] Disable host Samba service to prevent conflicts
|
||||
- [x] Configure UFW firewall for Samba ports
|
||||
- [x] Integrate Samba with Uptime Kuma (TCP port monitoring)
|
||||
- [x] Document Samba in CONTAINERS.md
|
||||
- [x] Fix Uptime Kuma network connectivity (multi-network bridge to all service networks)
|
||||
- [x] Deploy Gitea (Git repository hosting, ports 3002/2222)
|
||||
- [x] Configure Gitea with PostgreSQL database backend
|
||||
- [x] Configure NPM reverse proxy for https://git.schweitz.net with Let's Encrypt SSL
|
||||
- [x] Complete Gitea initial setup wizard and create admin account
|
||||
- [x] Integrate Gitea with Uptime Kuma (HTTP monitoring)
|
||||
- [x] Integrate Gitea with Organizr (dashboard tab)
|
||||
- [x] Document Gitea in CONTAINERS.md
|
||||
|
||||
### 🔄 In Progress
|
||||
|
||||
**Priority 1: Core-API Refactoring & Infrastructure Management:**
|
||||
### Priority 1: Core-API Refactoring & Infrastructure Management
|
||||
- [ ] **Code Cleanup:** Restructure Core API into function-specific controller files
|
||||
- [x] Create `/controllers` directory structure
|
||||
- [x] Create `/clients` directory structure
|
||||
- [x] Create `base.py` controller base class
|
||||
- [x] Add infrastructure settings to `config.py` (Portainer, NPM, Kuma URLs/credentials)
|
||||
- [x] Create credentials management system (credentials.py gitignored, credentials.example.py template)
|
||||
- [x] Add infrastructure settings to `config.py`
|
||||
- [x] Create credentials management system
|
||||
- [x] Update `main.py` routing to include infrastructure controller
|
||||
- [ ] Separate AI Orchestrator logic into `ai_controller.py`
|
||||
- [ ] Extract webscraper to `tools_controller.py`
|
||||
- [ ] Create `health_controller.py` for monitoring endpoints
|
||||
|
||||
- [x] **Infrastructure Management Controller:** Build automation API for service management
|
||||
- [x] Create `infrastructure_controller.py` with read/list endpoints
|
||||
- [x] **Portainer Integration:** HTTP client with access token authentication
|
||||
- [x] `get_endpoints()` - List Docker environments
|
||||
- [x] `get_stacks()` - List all stacks
|
||||
- [x] `get_stack(id)` - Get stack details
|
||||
- [x] `create_stack()` - Deploy from compose YAML
|
||||
- [x] `update_stack()` - Update existing stack
|
||||
- [x] `delete_stack()` - Remove stack
|
||||
- [x] **NPM Integration:** HTTP client with JWT bearer token + auto-refresh
|
||||
- [x] Token refresh mechanism (24h expiration handling)
|
||||
- [x] `get_proxy_hosts()` - List all proxy hosts
|
||||
- [x] `get_proxy_host(id)` - Get proxy details
|
||||
- [x] `create_proxy_host()` - Create new proxy configuration
|
||||
- [x] `get_certificates()` - List SSL certificates
|
||||
- [x] `create_certificate()` - Request Let's Encrypt cert
|
||||
- [x] **Read/List Endpoints Implemented & Tested:**
|
||||
- [x] `GET /infrastructure/health` - Check Portainer/NPM connectivity ✅ TESTED
|
||||
- [x] `GET /infrastructure/services` - List all deployed services ✅ TESTED (8 stacks)
|
||||
- [x] `GET /infrastructure/services/{name}` - Get service details ✅ TESTED
|
||||
- [x] `GET /infrastructure/ports` - List allocated ports (skeleton)
|
||||
- [x] `GET /infrastructure/domains` - List configured domains ✅ TESTED
|
||||
- [x] **Portainer API Token:** Generated programmatically and configured
|
||||
- [x] **Write Endpoints Implemented & Tested:**
|
||||
- [x] `POST /infrastructure/services` - Deploy new service ✅ TESTED
|
||||
- [x] `PUT /infrastructure/services/{name}` - Update service ✅ TESTED
|
||||
- [x] `DELETE /infrastructure/services/{name}` - Remove service ✅ TESTED
|
||||
- [x] `POST /infrastructure/proxy` - Create proxy host with SSL ✅ IMPLEMENTED
|
||||
- [ ] **Uptime Kuma Integration:** WebSocket client (deferred - complex Socket.IO)
|
||||
- [x] Portainer Integration (HTTP client with access token)
|
||||
- [x] NPM Integration (HTTP client with JWT bearer token + auto-refresh)
|
||||
- [x] Read/List Endpoints (all implemented & tested)
|
||||
- [x] Write Endpoints (POST/PUT/DELETE all implemented & tested)
|
||||
- [x] Portainer API Token generated programmatically
|
||||
- [ ] Uptime Kuma Integration (deferred - complex Socket.IO)
|
||||
- [ ] Replace ad-hoc shell scripts in `/stacks` with API endpoints
|
||||
- [ ] Add CLI wrapper for common operations
|
||||
|
||||
**Priority 2: AI Orchestrator Enhancement (Phase 2 - Memory Systems):**
|
||||
### Priority 2: AI Orchestrator Phase 2 (Memory Systems)
|
||||
- [ ] Implement Tier 1: ConversationBufferMemory (in-memory, last 10 turns)
|
||||
- [ ] Implement Tier 2: ConversationSummaryMemory (SQLite summaries)
|
||||
- [ ] Integrate Tier 3: VectorStoreRetrieverMemory (Qdrant semantic search)
|
||||
@@ -151,170 +43,65 @@
|
||||
- [ ] Add conversation history API endpoints
|
||||
- [ ] Test memory persistence across container restarts
|
||||
|
||||
### 📋 Planned (After Current Work)
|
||||
|
||||
**AI Orchestrator Phases 3-6:**
|
||||
- Phase 3: Multi-agent workflows with LangGraph (Router, Chat, Research, Code agents)
|
||||
- Phase 4: Tool integration (web search, web scrape, document search)
|
||||
- Phase 5: RAG & advanced memory (hybrid retrieval, document upload)
|
||||
- Phase 6: Production hardening (metrics, monitoring, optimization)
|
||||
|
||||
### 🔮 Backlog (Future Enhancements)
|
||||
|
||||
**Infrastructure Consolidation & Technical Debt:**
|
||||
1. **Centralized Database Container:** Consolidate SQLite databases from multiple services (Uptime Kuma, Organizr, etc.) into a single PostgreSQL/MySQL container for easier management and backups
|
||||
2. **Maintenance Container Consolidation:** Migrate maintenance container cron jobs into Core API endpoints with scheduled triggers - consolidate custom code into single service
|
||||
3. **Version Control Setup:** Initialize portainer-core repository in Gitea for proper version control, branching, and change tracking
|
||||
4. **Disaster Recovery Strategy:** Design offsite backup solution with restore/bootstrap scripts for full tower-of-joy recreation on new hardware
|
||||
|
||||
**Post-Phase 6 Integrations:**
|
||||
- Nextcloud integration (file search, calendar management)
|
||||
- ComfyUI integration (image generation)
|
||||
- Home Assistant integration (smart home control)
|
||||
- Custom mobile apps (iOS/Android)
|
||||
|
||||
## Current Blockers
|
||||
|
||||
None - All core services deployed and operational.
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Priority 1: Core-API Refactoring & Infrastructure Management
|
||||
|
||||
**Why Now:** Clean up technical debt before adding more features. Build proper infrastructure management API to eliminate ad-hoc scripts and enable programmatic service deployment.
|
||||
|
||||
**Immediate Actions:**
|
||||
1. **Refactor Core-API structure** - Create controller-based architecture for maintainability
|
||||
2. **Build Infrastructure Management API** - Automate Portainer/NPM/Kuma operations
|
||||
3. **Replace shell scripts** - Migrate `/stacks/*.sh` to proper API endpoints with CLI wrappers
|
||||
|
||||
**Benefits:**
|
||||
- Cleaner codebase for future AI Orchestrator development
|
||||
- Automated service deployment and monitoring setup
|
||||
- Programmatic infrastructure management (no more manual NPM/Kuma configuration)
|
||||
- Foundation for self-managing homelab
|
||||
|
||||
### Priority 2: AI Orchestrator Phase 2 (Memory Systems)
|
||||
|
||||
- **Implement 3-tier memory architecture:** ConversationBufferMemory (Tier 1), ConversationSummaryMemory (Tier 2), Qdrant VectorStore (Tier 3)
|
||||
- **Create Qdrant collections:** conversation_memory, documents, user_facts
|
||||
- **Build memory consolidation pipeline:** Automatic summarization and vector embedding
|
||||
- **Add conversation history endpoints:** Query and manage conversation memory
|
||||
- **Test integration with Open WebUI:** Verify memory persistence and recall
|
||||
|
||||
### Optional: Enhanced Capabilities
|
||||
|
||||
- **Connect Devices to Headscale:** Set up additional devices on mesh VPN for remote access
|
||||
- **Jellyfin Media Library:** Populate media libraries with content
|
||||
- **Nextcloud Desktop Clients:** Install sync clients on workstations
|
||||
- **External Monitoring:** Set up Uptime Kuma notifications (email, Discord, etc.)
|
||||
- **Advanced Automation:** Expand maintenance container with additional scheduled tasks
|
||||
|
||||
## Key Metrics
|
||||
|
||||
| Metric | Target | Current | Status |
|
||||
|--------|--------|---------|--------|
|
||||
| **Containers Running** | 15+ | 19 | 🟢 All Services Operational |
|
||||
| **GPU Accessible** | Yes | Yes | 🟢 Working |
|
||||
| **Storage Mounted** | 4.2TB | 3.6TB (58% used) | 🟢 Mounted |
|
||||
| **GPU Accessible** | Yes | Yes | 🟢 Working (RTX 2080 Ti) |
|
||||
| **Storage Used** | <80% | 58% HDD (3.6TB/3.7TB) | 🟢 Healthy |
|
||||
| **Services Accessible** | All | 19/19 | 🟢 Complete |
|
||||
| **Remote Access** | Working | Ready | 🟢 Headscale + NPM |
|
||||
| **Firewall Active** | Yes | Yes | 🟢 UFW Configured |
|
||||
| **Backups Configured** | Yes | Yes | 🟢 Maintenance Container |
|
||||
| **Cloud Storage** | Yes | Yes | 🟢 Nextcloud Deployed |
|
||||
| **File Sharing** | Yes | Yes | 🟢 Samba Deployed |
|
||||
| **AI Orchestrator** | Phase 6 | Phase 1 ✅ | 🟡 In Progress (Phase 2 next) |
|
||||
|
||||
## Version History
|
||||
|
||||
- **v0.7.1-gitea-deployment** (2025-11-14): Gitea Git service deployed with PostgreSQL, NPM reverse proxy (https://git.schweitz.net), SSH port 2222, full Uptime Kuma + Organizr integration
|
||||
- **v0.7.0-ai-orchestrator-phase1** (2025-11-13): AI Orchestrator Phase 1 complete - OpenAI-compatible API (`/v1/chat/completions`, `/v1/models`) with model aliasing and streaming support
|
||||
- **v0.6.0-applications** (2025-11-13): Nextcloud and Samba deployed - cloud storage, file sharing, multi-network Uptime Kuma integration
|
||||
- **v0.5.2-core-api** (2025-11-13): Core API deployed for Open WebUI functions, Python 3.12 upgrade (from EOL 3.8)
|
||||
- **v0.5.1-open-webui** (2025-11-12): Open WebUI deployed with built-in voice capabilities (local STT/TTS)
|
||||
- **v0.5.0-optimization** (2025-11-11): Phase 4 complete - Optimization & security (Watchtower, UFW, log rotation, maintenance container)
|
||||
- **v0.4.0-monitoring** (2025-11-11): Phase 3 complete - Monitoring stack deployed (Uptime Kuma, Netdata, Heimdall)
|
||||
- **v0.3.0-networking** (2025-11-11): Phase 2 complete - Headscale deployed with 10.99.0.0/16 mesh network
|
||||
- **v0.2.0-foundation** (2025-11-11): Phase 1 complete - Portainer, NPM, Ollama deployed with GPU support
|
||||
- **v0.1.0-planning** (2025-11-11): Project initialized, research and planning complete
|
||||
|
||||
---
|
||||
| **Backups Configured** | Yes | Yes | 🟢 Daily @ 3 AM |
|
||||
| **AI Orchestrator** | Phase 6 | Phase 1 ✅ | 🟡 Phase 2 In Progress |
|
||||
|
||||
## Quick Reference
|
||||
|
||||
**Documentation:**
|
||||
- Architecture research: `containers/research.md`
|
||||
- Implementation plan: `containers/implementation-plan.md`
|
||||
- Agent guidelines: `AGENTS.md`
|
||||
- System details: `SYSTEM.md`
|
||||
### Documentation
|
||||
- [Implementation Plans](PLANS.md) - Master plan tracker and roadmap
|
||||
- [Changelog](CHANGELOG.md) - Version history
|
||||
- [Container Reference](docs/reference/CONTAINERS.md) - All deployed services
|
||||
- [System Specifications](docs/reference/SYSTEM.md) - Hardware and software details
|
||||
- [Agent Guidelines](AGENTS.md) - Development conventions
|
||||
|
||||
**Key Paths:**
|
||||
- SSD configs: `/home/jpmschweitzer/docker-data/`
|
||||
- HDD content: `/mnt/media/`
|
||||
- Stacks: Managed in Portainer web UI
|
||||
### Key Paths
|
||||
- **SSD configs:** `/home/jpmschweitzer/docker-data/`
|
||||
- **HDD content:** `/mnt/media/`
|
||||
- **Stacks:** Managed in Portainer web UI
|
||||
- **Scripts:** `/mnt/media/Projects/portainer-core/scripts/`
|
||||
|
||||
**Active Services & Ports:**
|
||||
- **Portainer:** http://192.168.86.149:8001 (container management)
|
||||
- **Nginx Proxy Manager:** http://192.168.86.149:81 (reverse proxy admin)
|
||||
- **AMP:** http://192.168.86.149:8080 (game servers - native)
|
||||
### Active Services & URLs
|
||||
|
||||
**Infrastructure:**
|
||||
- **Portainer:** http://192.168.86.149:8080 (container management)
|
||||
- **Nginx Proxy Manager:** http://192.168.86.149:8000 (reverse proxy admin)
|
||||
- **Ollama:** http://192.168.86.149:11434 (ML models API)
|
||||
- **Headscale:** http://192.168.86.149:8085 (mesh VPN control server)
|
||||
- **Uptime Kuma:** http://192.168.86.149:3001 (service uptime monitoring)
|
||||
- **Netdata:** http://192.168.86.149:19999 (real-time system metrics)
|
||||
- **Heimdall:** http://192.168.86.149:8888 (unified dashboard)
|
||||
- **Watchtower:** (background service - automatic updates daily at 4 AM)
|
||||
- **Maintenance:** (background service - automated backups & scheduled tasks)
|
||||
|
||||
**Application Services:**
|
||||
- **Open WebUI:** http://192.168.86.149:82 (LLM chat interface)
|
||||
- **Core API:** http://192.168.86.149:8083 (OpenAPI functions for Open WebUI)
|
||||
- **Jellyfin:** http://192.168.86.149:8096 OR https://media.schweitz.net (GPU-accelerated media server)
|
||||
- **Nextcloud:** http://192.168.86.149:8082 OR https://cloud.schweitz.net (cloud storage & collaboration)
|
||||
- **Gitea:** http://192.168.86.149:3002 OR https://git.schweitz.net (Git repository hosting, SSH: port 2222)
|
||||
- **Samba:** \\\\192.168.86.149 or \\\\tower-of-joy (network file shares: Media, Downloads, Backups)
|
||||
**Networking:**
|
||||
- **Headscale:** http://192.168.86.149:8085 (mesh VPN control)
|
||||
|
||||
**Monitoring:**
|
||||
- **Uptime Kuma:** http://192.168.86.149:3001 (service monitoring)
|
||||
- **Netdata:** http://192.168.86.149:19999 (system metrics)
|
||||
- **Organizr:** http://192.168.86.149:8084 OR https://home.schweitz.net (unified dashboard)
|
||||
|
||||
**Applications:**
|
||||
- **Open WebUI:** http://192.168.86.149:8081 (LLM chat interface)
|
||||
- **Core API:** http://192.168.86.149:8083 (infrastructure management & AI orchestration)
|
||||
- **Jellyfin:** http://192.168.86.149:8096 OR https://media.schweitz.net (GPU media server)
|
||||
- **Nextcloud:** http://192.168.86.149:8082 OR https://cloud.schweitz.net (cloud storage)
|
||||
- **Gitea:** http://192.168.86.149:3002 OR https://git.schweitz.net (Git hosting, SSH: 2222)
|
||||
- **Samba:** \\\\192.168.86.149 or \\\\tower-of-joy (file shares: Media, Downloads, Backups)
|
||||
|
||||
**Background Services:**
|
||||
- **Watchtower:** Automatic updates daily @ 4 AM
|
||||
- **Maintenance:** Automated backups daily @ 3 AM
|
||||
|
||||
---
|
||||
|
||||
*Update this file as you complete each phase and checkpoint*
|
||||
|
||||
---
|
||||
|
||||
## Recent Updates
|
||||
|
||||
### 2025-11-14 Evening (Session 3)
|
||||
**Core-API Refactoring - Write Endpoints:**
|
||||
- ✅ Implemented POST /infrastructure/services - Deploy service from compose YAML
|
||||
- ✅ Implemented PUT /infrastructure/services/{name} - Update service configuration
|
||||
- ✅ Implemented DELETE /infrastructure/services/{name} - Remove service and stack
|
||||
- ✅ Implemented POST /infrastructure/proxy - Create proxy host with optional SSL
|
||||
- ✅ Tested all service management endpoints (POST/PUT/DELETE) with test-nginx stack
|
||||
- ✅ Updated main.py API description with write endpoints
|
||||
|
||||
**Status:** Infrastructure Management API complete (Phase 3 ✅)
|
||||
**Next:** Refactor existing controllers (Phase 4) or begin AI Orchestrator Phase 2
|
||||
|
||||
### 2025-11-14 Evening (Session 2)
|
||||
**Core-API Refactoring - Infrastructure Management:**
|
||||
- ✅ Created credentials management system (credentials.py gitignored)
|
||||
- ✅ Generated Portainer API token programmatically via API
|
||||
- ✅ Integrated infrastructure controller into main.py
|
||||
- ✅ Fixed Pydantic validation bug (status int→str conversion)
|
||||
- ✅ Tested all infrastructure read endpoints with live data
|
||||
- ✅ Verified 8 Portainer stacks detected
|
||||
- ✅ Domains endpoint working with SSL status
|
||||
|
||||
**Status:** Infrastructure read endpoints complete and tested
|
||||
**Next:** Implement write endpoints (POST/PUT/DELETE)
|
||||
|
||||
### 2025-11-13 Evening
|
||||
**Phase 1 Testing & Bug Fix:**
|
||||
- ✅ Completed comprehensive testing of Phase 1 implementation
|
||||
- ✅ Fixed model ID formatting issue (extra quotes in model names)
|
||||
- ✅ All 10/10 tests passing
|
||||
- ✅ Zero known issues remaining
|
||||
- ✅ Performance: 245ms average response time
|
||||
- ✅ 100% OpenAI API compatibility verified
|
||||
- ✅ Created comprehensive test results document (docs/phase1-test-results.md)
|
||||
|
||||
**Status:** Phase 1 100% complete and production-ready
|
||||
**Next:** Begin Phase 2 (Memory Systems) implementation
|
||||
*For detailed implementation history and completed work, see [CHANGELOG.md](CHANGELOG.md)*
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
# Maintenance Scripts Reference
|
||||
|
||||
Shell scripts for common maintenance tasks located in `/scripts/`.
|
||||
|
||||
## Available Scripts
|
||||
|
||||
| Script | Description | Usage |
|
||||
|--------|-------------|-------|
|
||||
| `gpu-check.sh` | Verify GPU passthrough in containers | `./scripts/gpu-check.sh` |
|
||||
| `health-check.sh` | Check all services and report status | `./scripts/health-check.sh` |
|
||||
| `setup-kuma-monitors.sh` | Manual guide for configuring Uptime Kuma monitors | `./scripts/setup-kuma-monitors.sh` |
|
||||
| `setup-kuma-monitors.py` | **Automated** Uptime Kuma monitor setup via API | `source .venv/bin/activate && python3 scripts/setup-kuma-monitors.py` |
|
||||
| `backup-configs.sh` | Backup all Docker configs | `./scripts/backup-configs.sh` |
|
||||
| `disk-usage.sh` | Report disk usage for SSD and HDD | `./scripts/disk-usage.sh` |
|
||||
| `update-stacks.sh` | Pull latest images and update containers | `./scripts/update-stacks.sh <stack-name>` |
|
||||
| `cleanup.sh` | Clean up unused Docker resources | `./scripts/cleanup.sh` |
|
||||
|
||||
## Making Scripts Executable
|
||||
|
||||
```bash
|
||||
# Make all scripts executable
|
||||
chmod +x scripts/*.sh
|
||||
|
||||
# Or individually
|
||||
chmod +x scripts/health-check.sh
|
||||
```
|
||||
|
||||
## Scheduling with Cron
|
||||
|
||||
Add to crontab for automated maintenance:
|
||||
|
||||
```bash
|
||||
# Edit crontab
|
||||
crontab -e
|
||||
|
||||
# Examples:
|
||||
# Daily health check at 8 AM
|
||||
0 8 * * * /home/jpmschweitzer/Projects/portainer-core/scripts/health-check.sh >> /var/log/portainer-core-health.log 2>&1
|
||||
|
||||
# Weekly cleanup on Sunday at 3 AM
|
||||
0 3 * * 0 /home/jpmschweitzer/Projects/portainer-core/scripts/cleanup.sh
|
||||
|
||||
# Daily backup at 2 AM
|
||||
0 2 * * * /home/jpmschweitzer/Projects/portainer-core/scripts/backup-configs.sh
|
||||
```
|
||||
|
||||
## Script Details
|
||||
|
||||
### GPU Check (`gpu-check.sh`)
|
||||
|
||||
Verifies GPU passthrough is working in GPU-enabled containers (Ollama, Jellyfin).
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/gpu-check.sh
|
||||
```
|
||||
|
||||
**Output:**
|
||||
- Lists all running containers with GPU access
|
||||
- Runs `nvidia-smi` inside each container
|
||||
- Reports any containers that fail GPU detection
|
||||
|
||||
### Health Check (`health-check.sh`)
|
||||
|
||||
Checks status of all deployed services and generates a health report.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/health-check.sh
|
||||
```
|
||||
|
||||
**Checks:**
|
||||
- Container running status
|
||||
- Container health status (if health check defined)
|
||||
- Port accessibility
|
||||
- Basic connectivity tests
|
||||
|
||||
### Uptime Kuma Monitor Setup
|
||||
|
||||
Two versions available:
|
||||
|
||||
**Manual Script (`setup-kuma-monitors.sh`):**
|
||||
- Interactive guide for adding monitors
|
||||
- Shows recommended settings for each service
|
||||
- Good for understanding monitor configuration
|
||||
|
||||
**Automated Script (`setup-kuma-monitors.py`):**
|
||||
- Python script using Uptime Kuma API
|
||||
- Automatically creates monitors for all services
|
||||
- Requires Uptime Kuma API key
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Automated setup
|
||||
source .venv/bin/activate
|
||||
python3 scripts/setup-kuma-monitors.py
|
||||
```
|
||||
|
||||
### Backup Configs (`backup-configs.sh`)
|
||||
|
||||
Backs up Docker container configurations and important data.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/backup-configs.sh
|
||||
```
|
||||
|
||||
**What it backs up:**
|
||||
- Docker Compose files from `/stacks/`
|
||||
- Container configs from `/home/jpmschweitzer/docker-data/`
|
||||
- Project documentation
|
||||
- Excludes large media files (those are backed up separately)
|
||||
|
||||
**Backup location:**
|
||||
- `/mnt/media/backups/portainer-core/`
|
||||
|
||||
See [Backup Procedures](../guides/backup-procedures.md) for comprehensive backup strategy.
|
||||
|
||||
### Disk Usage (`disk-usage.sh`)
|
||||
|
||||
Reports disk usage breakdown for SSD and HDD storage.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/disk-usage.sh
|
||||
```
|
||||
|
||||
**Output:**
|
||||
- Total SSD usage (`/home/jpmschweitzer/docker-data/`)
|
||||
- Total HDD usage (`/mnt/media/`)
|
||||
- Per-service breakdown
|
||||
- Available space warnings
|
||||
|
||||
### Update Stacks (`update-stacks.sh`)
|
||||
|
||||
Pulls latest images and updates a specific stack.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/update-stacks.sh <stack-name>
|
||||
|
||||
# Examples:
|
||||
./scripts/update-stacks.sh jellyfin
|
||||
./scripts/update-stacks.sh core-api
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Pulls latest images for the stack
|
||||
2. Stops containers gracefully
|
||||
3. Recreates containers with new images
|
||||
4. Removes old images
|
||||
5. Verifies containers started successfully
|
||||
|
||||
**Note:** Watchtower handles this automatically for most services. Use this script for manual updates or services excluded from Watchtower.
|
||||
|
||||
### Cleanup (`cleanup.sh`)
|
||||
|
||||
Cleans up unused Docker resources to free disk space.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/cleanup.sh
|
||||
```
|
||||
|
||||
**What it removes:**
|
||||
- Stopped containers
|
||||
- Unused images
|
||||
- Dangling build cache
|
||||
- Unused volumes (with confirmation prompt)
|
||||
- Unused networks
|
||||
|
||||
**Warning:** Always review what will be removed before confirming volume deletion.
|
||||
|
||||
## Script Guidelines
|
||||
|
||||
All scripts follow these conventions:
|
||||
|
||||
- Include error handling and exit codes
|
||||
- Use absolute paths for reliability
|
||||
- Log output for debugging
|
||||
- Exit with status codes (0 = success, non-zero = failure)
|
||||
- Include help text with `-h` or `--help` flags
|
||||
- Non-destructive by default (ask before deleting)
|
||||
|
||||
## Creating New Scripts
|
||||
|
||||
When adding new maintenance scripts:
|
||||
|
||||
1. Place in `/scripts/` directory
|
||||
2. Use `.sh` extension for shell scripts
|
||||
3. Make executable: `chmod +x scripts/your-script.sh`
|
||||
4. Add to this documentation
|
||||
5. Include help text and error handling
|
||||
6. Test thoroughly before scheduling with cron
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Backup Procedures](../guides/backup-procedures.md) - Comprehensive backup strategy
|
||||
- [Stacks Reference](stacks.md) - Stack deployment and management
|
||||
- [Automation Reference](AUTOMATION.md) - Portainer REST API automation
|
||||
@@ -0,0 +1,185 @@
|
||||
# Docker Compose Stacks Reference
|
||||
|
||||
Complete reference for all Docker Compose stacks in the portainer-core infrastructure.
|
||||
|
||||
## Deployment
|
||||
|
||||
See the [core-api OpenAPI documentation](http://localhost:8083/docs) for infrastructure management REST endpoints.
|
||||
|
||||
All stacks are located in the `/stacks/` directory and version-controlled.
|
||||
|
||||
## Stack Inventory
|
||||
|
||||
### Phase 1: Foundation
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Portainer** | `portainer.yml` | 8080, 8443 | No | Container management UI |
|
||||
| **Nginx Proxy Manager** | `nginx-proxy-manager.yml` | 8000, 80, 443 | No | Reverse proxy and unified web interface |
|
||||
| **Ollama** | `ollama.yml` | 11434 | **Yes** | ML model serving with GPU acceleration |
|
||||
|
||||
### Phase 2: Networking
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Headscale** | `headscale.yml` | 8085, 9090 | No | Self-hosted Tailscale control server |
|
||||
|
||||
### Phase 3: Monitoring
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Uptime Kuma** | `uptime-kuma.yml` | 3001 | No | Service availability monitoring |
|
||||
| **Netdata** | `netdata.yml` | 19999 | No | Real-time system performance monitoring |
|
||||
| **Heimdall** | `heimdall.yml` | 8888, 8889 | No | Application dashboard |
|
||||
|
||||
### Phase 4: Optimization
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Watchtower** | `watchtower.yml` | - | No | Automatic container updates |
|
||||
| **Duplicati** | `duplicati.yml` | 8200 | No | Backup solution |
|
||||
|
||||
### Applications
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Jellyfin** | `jellyfin.yml` | 8096, 8920, 7359, 1900 | **Yes** | Media server with GPU transcoding |
|
||||
| **Nextcloud** | `nextcloud.yml` | 8082 | No | Cloud storage (uses shared PostgreSQL and Redis) |
|
||||
| **Gitea** | `gitea.yml` | 3002, 2222 | No | Git repository hosting (includes PostgreSQL) |
|
||||
| **Samba** | `samba.yml` | 139, 445 | No | Network file sharing |
|
||||
| **Open WebUI** | `open-webui.yml` | 8081 | No | AI chat interface with Ollama integration |
|
||||
| **Core API** | `core-api.yml` | 8083 | No | Infrastructure management and AI orchestration |
|
||||
| **Qdrant** | `qdrant.yml` | 6333, 6334 | No | Vector database for embeddings |
|
||||
| **Organizr** | `organizr.yml` | 8084 | No | Unified dashboard |
|
||||
|
||||
### Shared Infrastructure
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **PostgreSQL Shared** | `postgres-shared.yml` | 5432 | No | Shared database for Nextcloud |
|
||||
| **Redis Shared** | `redis-shared.yml` | 6379 | No | Shared cache for Nextcloud |
|
||||
|
||||
## Port Allocation
|
||||
|
||||
### Infrastructure Services (8000-8099)
|
||||
- 8000: Nginx Proxy Manager (unified web interface)
|
||||
- 8080: Portainer
|
||||
- 8081: Open WebUI
|
||||
- 8082: Nextcloud
|
||||
- 8083: Core API
|
||||
- 8084: Organizr
|
||||
- 8085: Headscale
|
||||
- 8096: Jellyfin
|
||||
|
||||
### Git & Development Services
|
||||
- 2222: Gitea SSH
|
||||
- 3002: Gitea HTTP
|
||||
|
||||
### Monitoring Services (3000-3999, 19000-19999)
|
||||
- 3001: Uptime Kuma
|
||||
- 8200: Duplicati
|
||||
- 8888: Heimdall
|
||||
- 19999: Netdata
|
||||
|
||||
### ML/API Services (11000+)
|
||||
- 11434: Ollama
|
||||
- 6333: Qdrant HTTP
|
||||
- 6334: Qdrant gRPC
|
||||
|
||||
### Database Services
|
||||
- 5432: PostgreSQL (shared)
|
||||
- 6379: Redis (shared)
|
||||
|
||||
### Network Services
|
||||
- 80: HTTP (NPM reverse proxy)
|
||||
- 443: HTTPS (NPM reverse proxy)
|
||||
- 139, 445: Samba/SMB
|
||||
- 9090: Headscale metrics
|
||||
|
||||
## Storage Convention
|
||||
|
||||
All stacks follow the dual-disk strategy:
|
||||
|
||||
**SSD (Performance):**
|
||||
- Configs: `/home/jpmschweitzer/docker-data/<service>/config`
|
||||
- Cache: `/home/jpmschweitzer/docker-data/<service>/cache`
|
||||
- Databases: `/home/jpmschweitzer/docker-data/<service>/db`
|
||||
|
||||
**HDD (Capacity):**
|
||||
- User content: `/mnt/media/<service>/data`
|
||||
- Media files: `/mnt/media/<service>/media`
|
||||
- Backups: `/mnt/media/backups/<service>`
|
||||
|
||||
See [Shared Infrastructure Architecture](../architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md) for database and cache sharing details.
|
||||
|
||||
## GPU Services
|
||||
|
||||
Stacks requiring GPU access (marked with **Yes** above):
|
||||
- `ollama.yml` - ML model inference
|
||||
- `jellyfin.yml` - Hardware transcoding
|
||||
|
||||
**Prerequisites:**
|
||||
- NVIDIA Container Toolkit installed
|
||||
- GPU verified: `docker run --rm --gpus all nvidia/cuda:11.4.0-base-ubuntu20.04 nvidia-smi`
|
||||
|
||||
See [GPU Docker Configuration](../guides/gpu-docker-config.md) for setup details.
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
### Before Deploying
|
||||
|
||||
1. **Review environment variables** - Change default passwords!
|
||||
2. **Create directories** - Ensure volume paths exist
|
||||
3. **Check ports** - Verify no conflicts with existing services
|
||||
4. **GPU services** - Confirm NVIDIA toolkit installed
|
||||
5. **Update STATUS.md** - Plan the deployment
|
||||
|
||||
### After Deploying
|
||||
|
||||
1. **Test service** - Access web UI or API endpoint
|
||||
2. **Check logs** - `docker logs <container-name>`
|
||||
3. **Verify GPU** - `docker exec <container> nvidia-smi` (if applicable)
|
||||
4. **Update documentation** - Add to STATUS.md and CHANGELOG.md
|
||||
5. **Configure backup** - Add to Duplicati backup job
|
||||
6. **Add monitoring** - Configure Uptime Kuma checks
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Update a Stack
|
||||
|
||||
```bash
|
||||
# Pull latest images
|
||||
docker compose -f stacks/<stack-name>.yml pull
|
||||
|
||||
# Recreate containers with new images
|
||||
docker compose -f stacks/<stack-name>.yml up -d
|
||||
|
||||
# Or let Watchtower handle it automatically
|
||||
```
|
||||
|
||||
### Backup Stack Configuration
|
||||
|
||||
Stacks are version-controlled in the `/stacks/` directory. Backup container data separately using the backup procedures.
|
||||
|
||||
See [Backup Procedures](../guides/backup-procedures.md) for details.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
- Container won't start: `docker logs <container-name>`
|
||||
- Port conflicts: `sudo netstat -tulpn | grep <port>`
|
||||
- Permission issues: Check volume path ownership
|
||||
- GPU not detected: Verify NVIDIA toolkit and restart Docker
|
||||
|
||||
## Automation
|
||||
|
||||
The project includes automation scripts for stack management:
|
||||
|
||||
- `update-stack.sh` - Pull and update specific stack
|
||||
- See [Automation Reference](AUTOMATION.md) for Portainer REST API usage
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Container Reference](CONTAINERS.md) - Complete container profiles
|
||||
- [System Specifications](SYSTEM.md) - Hardware and software specs
|
||||
- [Shared Infrastructure Architecture](../architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md) - Database/cache sharing
|
||||
- [Maintenance Scripts](scripts.md) - Automated maintenance tasks
|
||||
@@ -0,0 +1,290 @@
|
||||
# Core API Service
|
||||
|
||||
OpenAPI-compatible functions for Open WebUI and infrastructure management, providing web scraping, AI orchestration, and Portainer automation capabilities.
|
||||
|
||||
## Features
|
||||
|
||||
### Web Scraper
|
||||
- Intelligent content extraction using Trafilatura
|
||||
- BeautifulSoup fallback for complex pages
|
||||
- Configurable content length limits
|
||||
- Optional link extraction
|
||||
- Perfect for feeding webpage content to LLMs
|
||||
|
||||
### Infrastructure Management
|
||||
- Portainer stack control (start/stop services)
|
||||
- Service status monitoring
|
||||
- Container health checks
|
||||
- Service group management
|
||||
- Read/write REST API
|
||||
|
||||
### AI Orchestration
|
||||
- OpenAI-compatible API endpoints
|
||||
- Model routing and management
|
||||
- Streaming responses
|
||||
- Function calling support
|
||||
- Multi-phase enhancement roadmap
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/
|
||||
├── config.py # Global application settings
|
||||
├── logging_config.py # Logging configuration
|
||||
├── base_schema.py # Base Pydantic models
|
||||
├── main.py # FastAPI application entry point
|
||||
└── modules/
|
||||
├── web_scraper/ # Web scraper module
|
||||
│ ├── config.py
|
||||
│ ├── schemas.py
|
||||
│ ├── service.py
|
||||
│ ├── router.py
|
||||
│ └── exceptions.py
|
||||
└── infrastructure/ # Infrastructure management
|
||||
├── config.py
|
||||
├── schemas.py
|
||||
├── service.py
|
||||
└── router.py
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Portainer Stack
|
||||
|
||||
1. Navigate to Portainer UI
|
||||
2. Go to **Stacks** → **Add Stack**
|
||||
3. Name: `core-api`
|
||||
4. Upload `stacks/core-api.yml` or paste contents
|
||||
5. Deploy
|
||||
|
||||
### Environment Variables
|
||||
|
||||
See `.env.example` in the service directory for all available configuration options.
|
||||
|
||||
Key variables:
|
||||
- `PORTAINER_URL` - Portainer API endpoint
|
||||
- `PORTAINER_API_KEY` - API key for Portainer authentication
|
||||
- `LOG_LEVEL` - Logging verbosity (DEBUG, INFO, WARNING, ERROR)
|
||||
- `CORS_ORIGINS` - Allowed CORS origins
|
||||
|
||||
## API Documentation
|
||||
|
||||
Once deployed, access documentation at:
|
||||
- **Swagger UI**: http://localhost:8083/docs
|
||||
- **ReDoc**: http://localhost:8083/redoc
|
||||
- **OpenAPI Spec**: http://localhost:8083/openapi.json
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Web Scraper
|
||||
|
||||
**POST /web-scraper/scrape**
|
||||
|
||||
Scrape and extract content from a website.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"url": "https://example.com/article",
|
||||
"extract_main_content": true,
|
||||
"include_links": false,
|
||||
"max_length": 10000
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"url": "https://example.com/article",
|
||||
"title": "Article Title",
|
||||
"content": "Extracted article content...",
|
||||
"extracted_at": "2025-11-12T19:30:00Z",
|
||||
"content_length": 5432,
|
||||
"links": null
|
||||
}
|
||||
```
|
||||
|
||||
### Infrastructure Management
|
||||
|
||||
**GET /infrastructure/services**
|
||||
|
||||
List all Portainer stacks with status.
|
||||
|
||||
Response:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "jellyfin",
|
||||
"status": "running",
|
||||
"containers": 1,
|
||||
"running_containers": 1
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**POST /infrastructure/services/{name}/start**
|
||||
|
||||
Start a service stack.
|
||||
|
||||
**POST /infrastructure/services/{name}/stop**
|
||||
|
||||
Stop a service stack.
|
||||
|
||||
**GET /infrastructure/service-groups**
|
||||
|
||||
Get service groupings and always-on services.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"service_groups": {
|
||||
"jellyfin": ["jellyfin"],
|
||||
"nextcloud": ["nextcloud"],
|
||||
"ai-stack": ["open-webui", "ollama", "qdrant"]
|
||||
},
|
||||
"always_on": ["portainer", "nginx-proxy-manager", "core-api"]
|
||||
}
|
||||
```
|
||||
|
||||
### Health Check
|
||||
|
||||
**GET /health**
|
||||
|
||||
Service health check endpoint.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"status": "healthy"
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with Open WebUI
|
||||
|
||||
### Method 1: Functions (OpenAPI Import)
|
||||
1. In Open WebUI, navigate to Functions
|
||||
2. Import from OpenAPI spec: `http://localhost:8083/openapi.json`
|
||||
3. Use functions directly in chat
|
||||
|
||||
### Method 2: Pipelines
|
||||
1. Create a pipeline that calls Core API endpoints
|
||||
2. Use as data source for LLM workflows
|
||||
|
||||
### Method 3: Direct API Calls
|
||||
```python
|
||||
import httpx
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post(
|
||||
"http://localhost:8083/web-scraper/scrape",
|
||||
json={
|
||||
"url": "https://example.com",
|
||||
"extract_main_content": True
|
||||
}
|
||||
)
|
||||
data = response.json()
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Requirements
|
||||
- Python 3.12+
|
||||
- Docker (for containerized deployment)
|
||||
|
||||
### Local Development
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Run locally
|
||||
uvicorn src.main:app --reload --host 0.0.0.0 --port 8083
|
||||
```
|
||||
|
||||
### Docker Build
|
||||
|
||||
```bash
|
||||
# Build image
|
||||
docker build -t core-api:latest .
|
||||
|
||||
# Run container
|
||||
docker run -p 8083:8083 core-api:latest
|
||||
```
|
||||
|
||||
## Logging
|
||||
|
||||
Logs are written to:
|
||||
- **Console**: stdout (captured by Docker)
|
||||
- **File**: `/app/logs/app.log` (persisted via volume mount)
|
||||
|
||||
Log format:
|
||||
```
|
||||
2025-11-12 19:30:00 | INFO | src.web_scraper.service:scrape_url:45 | Starting scrape for URL: https://example.com
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
- Runs as non-root user (uid 1000)
|
||||
- No authentication required (internal network only)
|
||||
- CORS configured for same-network access
|
||||
- Rate limiting: Not implemented (internal use only)
|
||||
- **Always-on service** - Cannot be stopped via infrastructure management
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
See [AI Orchestrator Plan](../../plans/active/ai-orchestrator-plan.md) for upcoming features:
|
||||
|
||||
### Phase 2: Memory Systems (In Progress)
|
||||
- Ephemeral, short-term, and long-term memory
|
||||
- Vector embeddings with Qdrant
|
||||
- Memory search and retrieval
|
||||
|
||||
### Phase 3: Multi-Model Management
|
||||
- Dynamic model routing
|
||||
- Cost optimization
|
||||
- Fallback strategies
|
||||
|
||||
### Phase 4: Reasoning & Chain-of-Thought
|
||||
- Structured reasoning
|
||||
- Multi-step problem solving
|
||||
- Verification and validation
|
||||
|
||||
### Phase 5: Agentic Workflows
|
||||
- Tool integration
|
||||
- Multi-agent orchestration
|
||||
- Autonomous task execution
|
||||
|
||||
### Phase 6: Production Optimization
|
||||
- Caching strategies
|
||||
- Performance tuning
|
||||
- Monitoring and metrics
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container won't start
|
||||
```bash
|
||||
docker logs core-api
|
||||
```
|
||||
|
||||
### API not responding
|
||||
```bash
|
||||
curl http://localhost:8083/health
|
||||
```
|
||||
|
||||
### Check OpenAPI spec
|
||||
```bash
|
||||
curl http://localhost:8083/openapi.json | jq
|
||||
```
|
||||
|
||||
### Portainer connection issues
|
||||
1. Verify `PORTAINER_URL` is correct
|
||||
2. Check `PORTAINER_API_KEY` is valid
|
||||
3. Ensure Portainer is accessible from core-api container
|
||||
4. Check Docker network connectivity
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Stacks Reference](../reference/stacks.md) - All Docker Compose stacks
|
||||
- [Automation Reference](../reference/AUTOMATION.md) - Portainer REST API details
|
||||
- [AI Orchestrator Plan](../../plans/active/ai-orchestrator-plan.md) - Feature roadmap
|
||||
- [Organizr Widget](organizr-widgets.md) - Service control UI integration
|
||||
@@ -0,0 +1,215 @@
|
||||
# Organizr Service Control Widget
|
||||
|
||||
A beautiful, responsive widget for managing on-demand services from your Organizr dashboard.
|
||||
|
||||
## Features
|
||||
|
||||
- ✨ **Real-time Status** - Live service status with container counts
|
||||
- 🎮 **One-Click Control** - Start/Stop services with a single click
|
||||
- 🔒 **Safety First** - Always-on services are protected and clearly marked
|
||||
- 🎨 **Beautiful UI** - Dark theme that matches Organizr
|
||||
- ⚡ **Auto-Refresh** - Updates every 10 seconds
|
||||
- 📱 **Responsive** - Works on desktop, tablet, and mobile
|
||||
|
||||
## Installation
|
||||
|
||||
### Method 1: Organizr Custom Homepage Item (Recommended)
|
||||
|
||||
1. **Copy the widget file** to a web-accessible location:
|
||||
```bash
|
||||
# If you have a web server serving files from /var/www/html:
|
||||
sudo cp organizr-widgets/service-control.html /var/www/html/widgets/
|
||||
|
||||
# Or use Organizr's public directory:
|
||||
cp organizr-widgets/service-control.html /path/to/organizr/plugins/widgets/
|
||||
```
|
||||
|
||||
2. **Add to Organizr Homepage**:
|
||||
- Open Organizr
|
||||
- Go to **Settings** → **Customize** → **Homepage Items**
|
||||
- Click **Add New Item**
|
||||
- Configure:
|
||||
- **Name**: "Service Control"
|
||||
- **Category**: Custom
|
||||
- **Type**: iFrame
|
||||
- **URL**: `http://localhost/widgets/service-control.html` (adjust path)
|
||||
- **Minimum Authentication**: User
|
||||
- **Enabled**: Yes
|
||||
- Save
|
||||
|
||||
3. **Add to Homepage**:
|
||||
- Go to **Settings** → **Customize** → **Appearance**
|
||||
- Edit your homepage layout
|
||||
- Add the "Service Control" item to desired location
|
||||
- Save
|
||||
|
||||
### Method 2: Organizr Custom HTML Tab
|
||||
|
||||
1. **Open Organizr Settings**:
|
||||
- Settings → **Tab Editor**
|
||||
|
||||
2. **Add New Tab**:
|
||||
- Click **Add Tab**
|
||||
- Configure:
|
||||
- **Tab Name**: "Services"
|
||||
- **Tab URL**: Leave empty
|
||||
- **Category**: Custom
|
||||
- **Type**: iFrame
|
||||
- **Image**: `images/tabs/services.png` (or your choice)
|
||||
|
||||
3. **Add Custom HTML**:
|
||||
- In the same tab configuration, find **Custom HTML** section
|
||||
- Copy and paste the entire contents of `service-control.html`
|
||||
- Save
|
||||
|
||||
4. **Access the Tab**:
|
||||
- The "Services" tab will now appear in your Organizr sidebar
|
||||
|
||||
### Method 3: Nginx Reverse Proxy Integration
|
||||
|
||||
If you want to serve the widget through Nginx Proxy Manager:
|
||||
|
||||
1. **Create a location** in your Organizr proxy host:
|
||||
```nginx
|
||||
location /widgets/ {
|
||||
alias /path/to/portainer-core/organizr-widgets/;
|
||||
autoindex off;
|
||||
}
|
||||
```
|
||||
|
||||
2. **Access via**: `https://your-organizr-domain.com/widgets/service-control.html`
|
||||
|
||||
## Configuration
|
||||
|
||||
### Changing API Endpoint
|
||||
|
||||
If your core-api is not on `localhost:8083`, edit the widget file:
|
||||
|
||||
```javascript
|
||||
const API_BASE = 'http://your-server:8083'; // Change this line
|
||||
```
|
||||
|
||||
### Adjusting Auto-Refresh Interval
|
||||
|
||||
Default is 10 seconds. To change:
|
||||
|
||||
```javascript
|
||||
setInterval(fetchServices, 10000); // Change 10000 to desired milliseconds
|
||||
```
|
||||
|
||||
### Customizing Displayed Services
|
||||
|
||||
By default, the widget shows all stoppable services (excludes always-on infrastructure).
|
||||
|
||||
To filter specific services, modify the `renderServices()` function:
|
||||
|
||||
```javascript
|
||||
const stoppableServices = services.filter(s =>
|
||||
!isAlwaysOn(s.name) &&
|
||||
['jellyfin', 'nextcloud', 'gitea', 'ai-stack'].includes(s.name) // Add this line
|
||||
);
|
||||
```
|
||||
|
||||
## Service Groups
|
||||
|
||||
The following service groups are defined (stopping one stops all in group):
|
||||
|
||||
- **jellyfin**: jellyfin
|
||||
- **nextcloud**: nextcloud (uses shared postgres-shared + redis-shared)
|
||||
- **gitea**: gitea, gitea-db
|
||||
- **ai-stack**: open-webui, ollama, qdrant
|
||||
- **samba**: samba
|
||||
|
||||
## Always-On Services (Cannot be stopped)
|
||||
|
||||
These infrastructure services are protected:
|
||||
- portainer
|
||||
- nginx-proxy-manager
|
||||
- core-api
|
||||
- uptime-kuma
|
||||
- organizr
|
||||
- headscale
|
||||
- watchtower
|
||||
- netdata
|
||||
- maintenance
|
||||
- postgres-shared (shared database infrastructure)
|
||||
- redis-shared (shared cache infrastructure)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Failed to connect to API"
|
||||
|
||||
**Problem**: Widget shows red error message
|
||||
|
||||
**Solutions**:
|
||||
1. Verify core-api is running: `docker ps | grep core-api`
|
||||
2. Check core-api URL is correct (localhost vs IP address)
|
||||
3. If accessing from remote, change `API_BASE` to full URL
|
||||
4. Check browser console for CORS errors
|
||||
|
||||
### CORS Issues
|
||||
|
||||
If accessing widget from a different domain than core-api:
|
||||
|
||||
**Option 1**: Update core-api CORS settings in `services/core-api/src/config.py`:
|
||||
```python
|
||||
cors_origins: list[str] = ["http://your-organizr-domain.com"]
|
||||
```
|
||||
|
||||
**Option 2**: Proxy the API through same domain using Nginx
|
||||
|
||||
### Services Not Appearing
|
||||
|
||||
**Check**:
|
||||
1. Services are deployed as Portainer stacks
|
||||
2. Services have proper labels: `com.docker.compose.project`
|
||||
3. Core-API can connect to Portainer
|
||||
4. Check browser console for errors
|
||||
|
||||
### Buttons Disabled
|
||||
|
||||
**Expected Behavior**:
|
||||
- Start button disabled when service is running
|
||||
- Stop button disabled when service is stopped
|
||||
- All buttons disabled for always-on services
|
||||
|
||||
## API Endpoints Used
|
||||
|
||||
The widget consumes these core-api endpoints:
|
||||
|
||||
- `GET /infrastructure/services` - Fetch service list with status
|
||||
- `GET /infrastructure/service-groups` - Fetch service groups and always-on list
|
||||
- `POST /infrastructure/services/{name}/start` - Start a service
|
||||
- `POST /infrastructure/services/{name}/stop` - Stop a service
|
||||
|
||||
See [Core API Documentation](core-api.md) for full API reference.
|
||||
|
||||
## Advanced Customization
|
||||
|
||||
### Colors
|
||||
|
||||
Edit the CSS variables in the `<style>` section:
|
||||
|
||||
```css
|
||||
.status-running {
|
||||
background: rgba(72, 187, 120, 0.2); /* Green background */
|
||||
color: #48bb78; /* Green text */
|
||||
}
|
||||
```
|
||||
|
||||
### Card Size
|
||||
|
||||
Adjust grid columns:
|
||||
|
||||
```css
|
||||
.service-grid {
|
||||
grid-template-columns: repeat(auto-fill, minmax(300px, 1fr));
|
||||
/* Change 300px to make cards wider/narrower */
|
||||
}
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Core API Service](core-api.md) - Infrastructure management API
|
||||
- [Stacks Reference](../reference/stacks.md) - All deployed services
|
||||
- [Automation Reference](../reference/AUTOMATION.md) - Portainer REST API
|
||||
Reference in New Issue
Block a user