restructure documentation
This commit is contained in:
@@ -0,0 +1,343 @@
|
||||
# Stack Automation Guide
|
||||
|
||||
## Overview
|
||||
|
||||
The `update-stack.sh` script enables programmatic stack updates via Portainer's REST API. This allows LLM agents (like Claude) and automation scripts to safely update Portainer stacks without requiring UI access.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Navigate to stacks directory
|
||||
cd /home/jpmschweitzer/Projects/portainer-core/stacks
|
||||
|
||||
# Update a stack (interactive mode - first time)
|
||||
./update-stack.sh open-webui.yml
|
||||
|
||||
# Subsequent updates (uses stored token)
|
||||
./update-stack.sh open-webui.yml
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### Authentication Flow
|
||||
|
||||
1. **First Run:**
|
||||
- Prompts for Portainer username/password
|
||||
- Authenticates with Portainer API
|
||||
- Generates JWT access token
|
||||
- Saves token to `.portainer-token` (gitignored)
|
||||
|
||||
2. **Subsequent Runs:**
|
||||
- Reads token from `.portainer-token`
|
||||
- Uses token for API calls
|
||||
- No credential prompts needed
|
||||
|
||||
### Update Process
|
||||
|
||||
1. Reads YAML file from `stacks/` directory
|
||||
2. Authenticates with Portainer (or uses cached token)
|
||||
3. Looks up stack by name (filename without .yml)
|
||||
4. Sends updated stack configuration via API
|
||||
5. Portainer validates and applies changes
|
||||
|
||||
## Usage Modes
|
||||
|
||||
### Interactive Mode (Human Operators)
|
||||
|
||||
```bash
|
||||
./update-stack.sh open-webui.yml
|
||||
```
|
||||
|
||||
**First run prompts for:**
|
||||
- Portainer username
|
||||
- Portainer password
|
||||
|
||||
**Token persists for subsequent runs.**
|
||||
|
||||
### Non-Interactive Mode (Automation/LLM Agents)
|
||||
|
||||
```bash
|
||||
export PORTAINER_USERNAME="admin"
|
||||
export PORTAINER_PASSWORD="your-secure-password"
|
||||
./update-stack.sh open-webui.yml
|
||||
```
|
||||
|
||||
**Use this mode for:**
|
||||
- CI/CD pipelines
|
||||
- LLM agent workflows
|
||||
- Automated deployment scripts
|
||||
- Cron jobs
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| `PORTAINER_URL` | No | `http://localhost:8080` | Portainer instance URL |
|
||||
| `PORTAINER_USERNAME` | Non-interactive only | - | Admin username |
|
||||
| `PORTAINER_PASSWORD` | Non-interactive only | - | Admin password |
|
||||
|
||||
## Examples
|
||||
|
||||
### Update Single Stack
|
||||
|
||||
```bash
|
||||
./update-stack.sh open-webui.yml
|
||||
```
|
||||
|
||||
### Update Multiple Stacks
|
||||
|
||||
```bash
|
||||
for stack in open-webui.yml ollama.yml core-api.yml; do
|
||||
./update-stack.sh "$stack"
|
||||
echo "---"
|
||||
done
|
||||
```
|
||||
|
||||
### LLM Agent Integration
|
||||
|
||||
```bash
|
||||
# Claude Code workflow example
|
||||
export PORTAINER_USERNAME="admin"
|
||||
export PORTAINER_PASSWORD="${PORTAINER_ADMIN_PASSWORD}" # from secure env
|
||||
|
||||
# Update stack after modifying YAML
|
||||
./update-stack.sh open-webui.yml
|
||||
|
||||
# Check result
|
||||
echo $? # 0 = success, 1 = failure
|
||||
```
|
||||
|
||||
### Remote Portainer Instance
|
||||
|
||||
```bash
|
||||
export PORTAINER_URL="https://portainer.example.com"
|
||||
./update-stack.sh my-stack.yml
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Token Storage
|
||||
|
||||
- Token stored in `.portainer-token` (gitignored)
|
||||
- File permissions: `600` (owner read/write only)
|
||||
- Token expires based on Portainer settings (default: 8 hours)
|
||||
- Re-authentication automatic if token expires
|
||||
|
||||
### Credentials
|
||||
|
||||
**DO NOT:**
|
||||
- ❌ Commit `.portainer-token` to git
|
||||
- ❌ Hardcode passwords in scripts
|
||||
- ❌ Share tokens between users
|
||||
- ❌ Use root/admin account for automation (create dedicated API user)
|
||||
|
||||
**DO:**
|
||||
- ✅ Use environment variables for non-interactive mode
|
||||
- ✅ Store credentials in secure password manager
|
||||
- ✅ Create dedicated Portainer user for automation
|
||||
- ✅ Rotate passwords regularly
|
||||
- ✅ Use `.gitignore` to exclude token file
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Create Automation User:**
|
||||
```
|
||||
Portainer → Users → Add User
|
||||
Username: portainer-automation
|
||||
Role: Environment Administrator (or custom)
|
||||
```
|
||||
|
||||
2. **Use Environment Variables:**
|
||||
```bash
|
||||
# In ~/.bashrc or secure environment
|
||||
export PORTAINER_USERNAME="portainer-automation"
|
||||
export PORTAINER_PASSWORD="$(pass show portainer/automation)" # from password manager
|
||||
```
|
||||
|
||||
3. **Restrict Permissions:**
|
||||
- Grant minimum required permissions
|
||||
- Limit to specific environments/stacks if possible
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Authentication Failed
|
||||
|
||||
```
|
||||
[ERROR] Failed to authenticate. Check credentials and try again.
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Verify username/password are correct
|
||||
- Check Portainer is accessible: `curl http://localhost:8080/api/status`
|
||||
- Ensure user has admin/environment admin role
|
||||
- Try removing `.portainer-token` and re-authenticating
|
||||
|
||||
### Stack Not Found
|
||||
|
||||
```
|
||||
[ERROR] Stack 'my-stack' not found in Portainer
|
||||
Available stacks:
|
||||
- open-webui
|
||||
- ollama
|
||||
- core-api
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Verify stack name matches filename (without .yml)
|
||||
- Check stack exists in Portainer UI
|
||||
- Stack name is case-sensitive
|
||||
- Create stack in Portainer first if it doesn't exist
|
||||
|
||||
### Connection Refused
|
||||
|
||||
```
|
||||
[ERROR] Failed to connect to Portainer at http://localhost:8080
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Check Portainer is running: `docker ps | grep portainer`
|
||||
- Verify port: Portainer default is 8080
|
||||
- Set `PORTAINER_URL` if using different port/host
|
||||
- Check firewall rules if accessing remote instance
|
||||
|
||||
### Token Expired
|
||||
|
||||
```
|
||||
[ERROR] Invalid authentication token
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Delete token file: `rm .portainer-token`
|
||||
- Re-run script to re-authenticate
|
||||
- Check Portainer token expiration settings
|
||||
|
||||
### YAML Validation Error
|
||||
|
||||
```
|
||||
[ERROR] Stack update failed: invalid compose file
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Validate YAML syntax: `yamllint open-webui.yml`
|
||||
- Check Docker Compose version compatibility
|
||||
- Review Portainer logs: `docker logs portainer`
|
||||
- Test with `docker compose config -f open-webui.yml`
|
||||
|
||||
## Integration with LLM Agents
|
||||
|
||||
### Claude Code Workflow
|
||||
|
||||
This script is designed to integrate seamlessly with Claude Code workflows:
|
||||
|
||||
1. **Agent modifies YAML file:**
|
||||
```python
|
||||
# Claude uses Edit tool to update open-webui.yml
|
||||
```
|
||||
|
||||
2. **Agent calls update script:**
|
||||
```bash
|
||||
cd /home/jpmschweitzer/Projects/portainer-core/stacks
|
||||
./update-stack.sh open-webui.yml
|
||||
```
|
||||
|
||||
3. **Agent verifies deployment:**
|
||||
```bash
|
||||
docker logs open-webui --tail 20
|
||||
curl http://localhost:82 # Verify service
|
||||
```
|
||||
|
||||
### Setting Up for Claude
|
||||
|
||||
Add to user profile or environment:
|
||||
|
||||
```bash
|
||||
# In ~/.bashrc or secure location
|
||||
export PORTAINER_USERNAME="admin"
|
||||
export PORTAINER_PASSWORD="your-secure-password"
|
||||
|
||||
# Or use password manager
|
||||
export PORTAINER_PASSWORD="$(pass show portainer/admin)"
|
||||
```
|
||||
|
||||
Then Claude can directly call:
|
||||
```bash
|
||||
./update-stack.sh <stack-file.yml>
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Custom Portainer URL
|
||||
|
||||
```bash
|
||||
# Connect to remote Portainer
|
||||
export PORTAINER_URL="https://portainer.mydomain.com"
|
||||
./update-stack.sh open-webui.yml
|
||||
```
|
||||
|
||||
### Token Management
|
||||
|
||||
```bash
|
||||
# View current token (for debugging)
|
||||
cat .portainer-token | base64 -d | jq
|
||||
|
||||
# Force re-authentication
|
||||
rm .portainer-token
|
||||
./update-stack.sh open-webui.yml
|
||||
|
||||
# Use specific token
|
||||
echo "your-jwt-token-here" > .portainer-token
|
||||
chmod 600 .portainer-token
|
||||
```
|
||||
|
||||
### Dry Run (Check Only)
|
||||
|
||||
```bash
|
||||
# Validate YAML before updating
|
||||
docker compose -f open-webui.yml config
|
||||
|
||||
# Check current stack status
|
||||
curl -s http://localhost:8080/api/stacks \
|
||||
-H "Authorization: Bearer $(cat .portainer-token)" | jq
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
The script uses these Portainer API endpoints:
|
||||
|
||||
- `POST /api/auth` - Authenticate and get token
|
||||
- `GET /api/endpoints` - List Docker endpoints
|
||||
- `GET /api/stacks` - List all stacks
|
||||
- `PUT /api/stacks/{id}` - Update specific stack
|
||||
|
||||
For full API documentation: https://docs.portainer.io/api/docs
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Regular Tasks
|
||||
|
||||
- **Monthly:** Rotate automation user password
|
||||
- **Quarterly:** Review and audit API access logs
|
||||
- **After incidents:** Revoke and regenerate tokens
|
||||
|
||||
### Token Rotation
|
||||
|
||||
```bash
|
||||
# Revoke old token (Portainer UI)
|
||||
Portainer → Users → [user] → Access Tokens → Revoke All
|
||||
|
||||
# Re-authenticate
|
||||
rm .portainer-token
|
||||
./update-stack.sh open-webui.yml
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check Portainer logs: `docker logs portainer`
|
||||
2. Review this guide's Troubleshooting section
|
||||
3. Check Portainer API docs: https://docs.portainer.io/api/docs
|
||||
4. Open issue in project repository
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-11-14*
|
||||
@@ -0,0 +1,609 @@
|
||||
# Container Reference Guide
|
||||
|
||||
> Documentation of all deployed containers in the tower-of-joy infrastructure
|
||||
>
|
||||
> Last Updated: 2025-11-16
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure Layer
|
||||
|
||||
### Portainer
|
||||
|
||||
Portainer provides the web-based container management interface for the entire stack, offering visual control over Docker containers, stacks, images, volumes, and networks. It serves as the primary management tool for deploying and monitoring all other services, with GPU device management enabled for allocation to ML and transcoding workloads. The interface replaces the need for manual Docker CLI operations and provides real-time container logs, stats, and control.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `portainer/portainer-ce:latest` |
|
||||
| **Container Name** | `portainer` |
|
||||
| **Access URL** | http://192.168.86.149:8001 |
|
||||
| **External Access** | LAN only (behind firewall) |
|
||||
| **Port Mapping** | 8001:9000 (HTTP), 8443:9443 (HTTPS) |
|
||||
| **Network Mode** | Host |
|
||||
| **Restart Policy** | `always` |
|
||||
| **Volume Mounts** | `portainer_data:/data`, `/var/run/docker.sock` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | None (base service) |
|
||||
|
||||
---
|
||||
|
||||
### PostgreSQL Shared
|
||||
|
||||
PostgreSQL Shared is a centralized PostgreSQL 17 database server providing isolated database instances for multiple applications across the infrastructure, including Authentik (SSO), Gitea (Git hosting), and future services requiring relational database storage. It implements a shared infrastructure pattern where each application gets its own database and user credentials while sharing the same PostgreSQL instance for resource efficiency. The service stores all database data on the HDD with automated backups scheduled to the backups directory, providing persistent storage with volume-based data retention across container updates.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `postgres:17` |
|
||||
| **Container Name** | `postgres-shared` |
|
||||
| **Access URL** | N/A (internal database server) |
|
||||
| **External Access** | No (docker-dataplane network only) |
|
||||
| **Port Mapping** | 5432:5432 (PostgreSQL) |
|
||||
| **Network Mode** | Bridge (docker-dataplane) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `/home/jpmschweitzer/docker-data/postgres-shared/data:/var/lib/postgresql/data`, `/home/jpmschweitzer/docker-data/postgres-shared/backups:/backups` |
|
||||
| **Environment** | `POSTGRES_PASSWORD=<secure-password>`, `POSTGRES_DB=postgres`, `TZ=Europe/Amsterdam`, `PGDATA=/var/lib/postgresql/data/pgdata` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | docker-dataplane network |
|
||||
| **Databases** | `authentik` (Authentik SSO), `gitea` (Git hosting), `postgres` (default/admin) |
|
||||
| **Database Users** | `authentik_user`, `gitea_user`, `postgres` (superuser) |
|
||||
| **Health Check** | `pg_isready -U postgres` (30s interval) |
|
||||
| **Backup Strategy** | `/backups` volume for pg_dump exports |
|
||||
|
||||
**Initialization**: Databases and users are created manually after first deployment:
|
||||
```bash
|
||||
docker exec -i postgres-shared psql -U postgres <<'EOF'
|
||||
CREATE DATABASE authentik;
|
||||
CREATE USER authentik_user WITH PASSWORD '<password>';
|
||||
GRANT ALL PRIVILEGES ON DATABASE authentik TO authentik_user;
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Redis Shared
|
||||
|
||||
Redis Shared is a centralized Redis 7 key-value store providing cache, session storage, and message queue capabilities for multiple applications, with logical database isolation (DB 0-15) allowing each service to maintain separate keyspaces within the same Redis instance. It implements a shared infrastructure pattern where applications like Authentik use DB 0 for sessions/cache while future services can use DB 1-15, eliminating the need for separate Redis containers per application. The service stores data on the HDD for persistence across restarts, with AOF (Append-Only File) enabled for durability and optional RDB snapshots for backup points.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `redis:7-alpine` |
|
||||
| **Container Name** | `redis-shared` |
|
||||
| **Access URL** | N/A (internal cache server) |
|
||||
| **External Access** | No (docker-dataplane network only) |
|
||||
| **Port Mapping** | 6379:6379 (Redis) |
|
||||
| **Network Mode** | Bridge (docker-dataplane) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `/home/jpmschweitzer/docker-data/redis-shared/data:/data` |
|
||||
| **Command** | `redis-server --appendonly yes --dir /data` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | docker-dataplane network |
|
||||
| **Database Allocation** | DB 0: Authentik, DB 1-15: Available for future services |
|
||||
| **Persistence** | AOF (Append-Only File) enabled for durability |
|
||||
| **Health Check** | `redis-cli ping` returns PONG (30s interval) |
|
||||
| **Connection String** | `redis://redis-shared:6379/0` (DB 0), `redis://redis-shared:6379/1` (DB 1), etc. |
|
||||
|
||||
---
|
||||
|
||||
### Nginx Proxy Manager
|
||||
|
||||
Nginx Proxy Manager serves as the unified reverse proxy and SSL certificate manager, providing a web-based interface for routing HTTP/HTTPS traffic to backend services with automatic Let's Encrypt certificate provisioning. It consolidates access to all web services through a single entry point with path-based or subdomain routing, eliminating the need to remember individual service ports. The service handles SSL termination, proxy host configuration, and access list management through an intuitive dashboard.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `jc21/nginx-proxy-manager:latest` |
|
||||
| **Container Name** | `nginx-proxy-manager` |
|
||||
| **Access URL** | http://192.168.86.149:81 |
|
||||
| **External Access** | LAN + Internet (ports 80/443 forwarded) |
|
||||
| **Port Mapping** | 81:81 (Admin), 80:80 (HTTP), 443:443 (HTTPS) |
|
||||
| **Network Mode** | Host |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/nginx-proxy-manager/data:/data`, `~/docker-data/nginx-proxy-manager/letsencrypt:/etc/letsencrypt` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | None (reverse proxy for other services) |
|
||||
|
||||
---
|
||||
|
||||
### Ollama
|
||||
|
||||
Ollama is a GPU-accelerated large language model server that provides a REST API for running local LLM inference with models up to 13B parameters, leveraging the RTX 2080 Ti's 11GB VRAM for fast on-device AI capabilities. It manages model downloads, quantization, and serving through a simple API compatible with OpenAI's format, supporting use cases like code generation, chat applications, and text processing without cloud dependencies. The service stores models on the SSD for quick loading times and maintains persistent model storage across container restarts.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `ollama/ollama:latest` |
|
||||
| **Container Name** | `ollama` |
|
||||
| **Access URL** | http://192.168.86.149:11434 |
|
||||
| **External Access** | LAN only (API endpoint) |
|
||||
| **Port Mapping** | 11434:11434 (API) |
|
||||
| **Network Mode** | Bridge (custom network) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/ollama/models:/root/.ollama` |
|
||||
| **Resource Limits** | Memory: 8GB, GPU: 11GB VRAM |
|
||||
| **GPU Required** | Yes (NVIDIA RTX 2080 Ti) |
|
||||
| **GPU Configuration** | `NVIDIA_VISIBLE_DEVICES=all`, `NVIDIA_DRIVER_CAPABILITIES=compute,utility` |
|
||||
| **Dependencies** | NVIDIA Container Toolkit |
|
||||
| **Typical Models** | llama3.2:3b (~2GB), mistral:7b (~4GB), codellama:7b (~4GB) |
|
||||
|
||||
---
|
||||
|
||||
### Code-Server
|
||||
|
||||
Code-Server provides a browser-based Visual Studio Code IDE running directly on the host system, offering a persistent development environment with full access to host-level configurations, filesystems, and systemd services without the limitations of containerization. It replaces traditional SSH access by providing a rich IDE experience that survives network disconnections through session persistence, with integrated terminal access, file explorer, git integration, and extension support. The service runs as a systemd service on the host, listening on localhost and exposed externally through Nginx Proxy Manager with SSL encryption and multi-layer authentication for secure remote development access.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Deployment Type** | Host-based systemd service (NOT containerized) |
|
||||
| **Binary Location** | `/usr/bin/code-server` |
|
||||
| **Service Name** | `code-server.service` |
|
||||
| **Access URL (LAN)** | http://127.0.0.1:8084 (localhost only) |
|
||||
| **Access URL (Public)** | https://code.schweitz.net |
|
||||
| **External Access** | Yes (via NPM reverse proxy with SSL) |
|
||||
| **Port Binding** | 127.0.0.1:8084 (not exposed to network) |
|
||||
| **User/Group** | `jpmschweitzer:jpmschweitzer` |
|
||||
| **Restart Policy** | `always` (systemd) |
|
||||
| **Config Location** | `~/.config/code-server/config.yaml` |
|
||||
| **Data Storage (SSD)** | `~/docker-data/code-server/user-data/` (settings, workspace), `~/docker-data/code-server/extensions/` (extensions) |
|
||||
| **Resource Limits** | None (native host process) |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | NPM (reverse proxy), systemd |
|
||||
| **Authentication** | Triple-layer: NPM access list, code-server password, SSL certificate |
|
||||
| **WebSocket Support** | Required (enabled via NPM proxy) |
|
||||
| **Typical Memory** | ~200-500MB (depends on workspace size) |
|
||||
| **Setup Guide** | `docs/code-server-setup.md` |
|
||||
|
||||
---
|
||||
|
||||
## Networking Layer
|
||||
|
||||
### Headscale
|
||||
|
||||
Headscale is a self-hosted control server for Tailscale's mesh VPN protocol, creating a private software-defined network across all connected devices with end-to-end encryption and zero-configuration NAT traversal. It enables secure remote access to all homelab services from anywhere without exposing ports to the internet, using a custom 10.99.0.0/16 IP range for the mesh network. The service manages device registration, authentication, and mesh routing while maintaining full data sovereignty compared to the hosted Tailscale control plane.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `headscale/headscale:latest` |
|
||||
| **Container Name** | `headscale` |
|
||||
| **Access URL** | http://192.168.86.149:8085 |
|
||||
| **External Access** | LAN only (control server) |
|
||||
| **Port Mapping** | 8085:8080 (Web/API), 9090:9090 (Metrics) |
|
||||
| **Network Mode** | Bridge (custom network) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/headscale/config:/etc/headscale`, `~/docker-data/headscale/data:/var/lib/headscale` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | None |
|
||||
| **Network Range** | 10.99.0.0/16 (mesh IPs) |
|
||||
| **Pre-Auth Keys** | 24-hour expiration |
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Layer
|
||||
|
||||
### Uptime Kuma
|
||||
|
||||
Uptime Kuma monitors the availability and response times of all infrastructure and application services, providing real-time status dashboards with historical uptime tracking, incident detection, and notification capabilities. It performs HTTP, TCP, and ICMP checks at configurable intervals against each service endpoint, alerting on downtime events through multiple notification channels including email, Discord, and Slack. The service maintains a SQLite database of uptime history and response time metrics accessible through a clean web interface.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `louislam/uptime-kuma:latest` |
|
||||
| **Container Name** | `uptime-kuma` |
|
||||
| **Access URL** | http://192.168.86.149:3001 |
|
||||
| **External Access** | LAN only (monitoring dashboard) |
|
||||
| **Port Mapping** | 3001:3001 (Web UI) |
|
||||
| **Network Mode** | Bridge |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/uptime-kuma:/app/data` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | None (monitors other services) |
|
||||
| **Database** | SQLite (persistent in volume) |
|
||||
| **Check Intervals** | 60 seconds (configurable) |
|
||||
|
||||
---
|
||||
|
||||
### Netdata
|
||||
|
||||
Netdata provides comprehensive real-time system performance monitoring with per-second metric collection for CPU, RAM, disk I/O, network traffic, and Docker container resource usage, displaying everything through interactive web dashboards with zero configuration required. It collects thousands of metrics automatically with minimal overhead, offering drill-down capabilities from system-wide views to per-container and per-process analysis. The service maintains short-term metric history in RAM and can stream data to long-term storage backends for historical analysis.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `netdata/netdata:latest` |
|
||||
| **Container Name** | `netdata` |
|
||||
| **Access URL** | http://192.168.86.149:19999 |
|
||||
| **External Access** | LAN only (metrics dashboard) |
|
||||
| **Port Mapping** | 19999:19999 (Web UI) |
|
||||
| **Network Mode** | Host (for full system visibility) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `/proc:/host/proc:ro`, `/sys:/host/sys:ro`, `/var/run/docker.sock:/var/run/docker.sock:ro` |
|
||||
| **Capabilities** | `SYS_PTRACE`, `apparmor:unconfined` |
|
||||
| **Resource Limits** | None (monitoring overhead ~1-3% CPU) |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | Docker socket (read-only) |
|
||||
| **Metric Retention** | ~1 hour (RAM-based) |
|
||||
|
||||
---
|
||||
|
||||
### Organizr
|
||||
|
||||
Organizr serves as a comprehensive unified dashboard that consolidates all homelab services into a single tabbed interface with integrated homepage widgets showing real-time statistics from Jellyfin streams, Netdata metrics, Uptime Kuma status checks, and download client activity. It provides customizable authentication per-tab with support for SSO integration, user management with group-based access control, and a mobile-responsive interface for managing the entire infrastructure from anywhere. The service acts as a central hub replacing the need for multiple bookmarks or remembering service ports, offering both quick-access tabs and homepage cards with live data feeds from connected services.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `organizr/organizr:latest` |
|
||||
| **Container Name** | `organizr` |
|
||||
| **Access URL (LAN)** | http://192.168.86.149:9999 |
|
||||
| **Access URL (Public)** | https://home.schweitz.net |
|
||||
| **External Access** | Yes (via NPM reverse proxy with SSL) |
|
||||
| **Port Mapping** | 9999:80 (HTTP), 443:443 (HTTPS) |
|
||||
| **Network Mode** | Bridge |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/organizr:/config` |
|
||||
| **Environment** | Built-in (no custom env vars) |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | None (integrates with other services via API) |
|
||||
| **Database** | SQLite at `/config/db/organizrDashboardDb.db` |
|
||||
| **Database Size** | ~5-10MB (typical) |
|
||||
| **Integrated Services** | Jellyfin, Netdata, Uptime Kuma |
|
||||
| **Authentication** | Internal (supports SSO, Plex OAuth, LDAP) |
|
||||
|
||||
---
|
||||
|
||||
## Optimization Layer
|
||||
|
||||
### Watchtower
|
||||
|
||||
Watchtower automatically monitors all running containers for updated images and performs rolling updates on a configurable schedule, ensuring the infrastructure stays current with security patches and feature releases without manual intervention. It checks Docker Hub and configured registries daily at 4 AM, pulls new images when available, gracefully stops containers, deploys updated versions, and cleans up old images to prevent disk bloat. The service logs all update activities and can send notifications through various channels when updates occur.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `containrrr/watchtower:latest` |
|
||||
| **Container Name** | `watchtower` |
|
||||
| **Access URL** | N/A (background service) |
|
||||
| **External Access** | N/A |
|
||||
| **Port Mapping** | None (no exposed ports) |
|
||||
| **Network Mode** | Bridge |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `/var/run/docker.sock:/var/run/docker.sock` |
|
||||
| **Environment** | `WATCHTOWER_CLEANUP=true`, `WATCHTOWER_SCHEDULE=0 0 4 * * *` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | Docker socket (read-write for updates) |
|
||||
| **Schedule** | Daily at 4:00 AM |
|
||||
| **Cleanup** | Automatic (removes old images) |
|
||||
|
||||
---
|
||||
|
||||
### Maintenance Container
|
||||
|
||||
The maintenance container runs scheduled automation tasks including nightly Docker configuration backups with 30-day retention, disk space monitoring, log cleanup, and future expansion for health checks and system maintenance scripts. It executes cron-based jobs at 3 AM daily to archive all Docker Compose configurations, container settings, and persistent data to the backup directory on the HDD with timestamped snapshots. The container provides a centralized location for all homelab automation without cluttering the host system with multiple cron entries.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `alpine:latest` |
|
||||
| **Container Name** | `maintenance` |
|
||||
| **Access URL** | N/A (background service) |
|
||||
| **External Access** | N/A |
|
||||
| **Port Mapping** | None (no exposed ports) |
|
||||
| **Network Mode** | Bridge |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data:/source:ro`, `/mnt/media/backups:/backups` |
|
||||
| **Command** | Runs crond with custom crontab |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | None |
|
||||
| **Schedule** | Daily at 3:00 AM (backups) |
|
||||
| **Backup Retention** | 30 days |
|
||||
| **Backup Size** | ~94MB per snapshot |
|
||||
|
||||
---
|
||||
|
||||
## Application Layer
|
||||
|
||||
### Open WebUI
|
||||
|
||||
Open WebUI is a feature-rich, self-hosted web interface for interacting with large language models via Ollama, providing a ChatGPT-like experience with support for multiple models, conversation history, RAG (Retrieval-Augmented Generation), web search integration, and user authentication. It offers a modern chat interface with streaming responses, markdown rendering, code syntax highlighting, and conversation management, enabling seamless switching between different LLM models and maintaining persistent chat histories in a local database. The service integrates directly with the local Ollama instance for GPU-accelerated inference without cloud dependencies, supporting features like web search via DuckDuckGo, document upload for context, and multi-user access with authentication. It is documented at: https://docs.openwebui.com/
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `ghcr.io/open-webui/open-webui:main` |
|
||||
| **Container Name** | `open-webui` |
|
||||
| **Access URL** | http://192.168.86.149:82 |
|
||||
| **External Access** | LAN only (not yet proxied) |
|
||||
| **Port Mapping** | 82:8080 (HTTP) |
|
||||
| **Network Mode** | Bridge (custom network: ai-network) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/open-webui:/app/backend/data` |
|
||||
| **Environment** | `OLLAMA_BASE_URL=http://192.168.86.149:11434`, `DEFAULT_MODELS=llama3.2:3b`, `ENABLE_RAG_WEB_SEARCH=true`, `ENABLE_OLLAMA_API=true`, `WEBUI_AUTH=true`, `RAG_WEB_SEARCH_ENGINE=duckduckgo`, `AUDIO_STT_ENGINE=openai`, `AUDIO_TTS_ENGINE=openai` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No (uses Ollama for GPU inference) |
|
||||
| **Dependencies** | Ollama (ML inference backend) |
|
||||
| **Database** | SQLite (persistent in volume) |
|
||||
| **Authentication** | Built-in user management |
|
||||
| **Integrated Services** | Ollama, DuckDuckGo (web search) |
|
||||
|
||||
---
|
||||
|
||||
### Core API
|
||||
|
||||
Core API provides OpenAI-compatible HTTP functions for Open WebUI, extending LLM capabilities with AI orchestration, web scraping, and content processing services in a hot-reload development environment. The service implements **Phase 1 of the AI Orchestrator** plan, providing `/v1/chat/completions` and `/v1/models` endpoints with full OpenAI API compatibility, model aliasing (gpt-3.5-turbo → gemma:7b), and streaming support via Server-Sent Events. It uses Trafilatura for intelligent content extraction with BeautifulSoup fallback, offering configurable content length limits and optional link extraction optimized for feeding webpage content to language models. The service runs on Python 3.12 with mounted source code for instant updates, maintaining a persistent venv in docker-data for fast container restarts and development agility.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `python:3.12` |
|
||||
| **Container Name** | `core-api` |
|
||||
| **Access URL** | http://192.168.86.149:8083 |
|
||||
| **API Documentation** | http://192.168.86.149:8083/docs (Swagger UI) |
|
||||
| **External Access** | LAN only (internal API) |
|
||||
| **Port Mapping** | 8083:8083 (HTTP) |
|
||||
| **Network Mode** | Bridge (custom network: ai-dataplane) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `/home/jpmschweitzer/Projects/portainer-core/services/core-api:/app` (source code), `~/docker-data/core-api/venv:/venv` (dependencies), `~/docker-data/core-api/logs:/app/logs` (logs) |
|
||||
| **Environment** | `APP_NAME=Core API`, `APP_VERSION=1.0.0-phase1`, `DEBUG=true`, `PORT=8083`, `LOG_LEVEL=INFO`, `PYTHONPATH=/app`, Model aliases: `ALIAS_GPT35=gemma:7b`, `ALIAS_GPT4=mistral:7b` |
|
||||
| **Command** | Hot-reload with uvicorn: `--reload --reload-dir /app/src` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No (proxies requests to Ollama which uses GPU) |
|
||||
| **Dependencies** | Ollama (model inference), Open WebUI (consumes this API), ai-dataplane network |
|
||||
| **Framework** | FastAPI 0.115.0, Uvicorn 0.32.0, Pydantic 2.10.4, httpx 0.28.1 |
|
||||
| **Key Features** | OpenAI-compatible API (`/v1/chat/completions`, `/v1/models`), Model aliasing (OpenAI → local models), Streaming & non-streaming responses, Web scraping (Trafilatura, BeautifulSoup), Hot-reload development, OpenAPI spec, Infrastructure management (Portainer, Uptime Kuma) |
|
||||
| **AI Orchestrator** | **Phase 1 Complete** - OpenAI API wrapper with model routing. Phase 2+ will add memory systems, multi-agent workflows, and tool integration. |
|
||||
| **Health Check** | `GET /health` (30s interval) - checks API status and Ollama connectivity |
|
||||
| **Monitoring API** | Full CRUD for Uptime Kuma monitors via Socket.IO: `GET /infrastructure/monitors`, `POST /infrastructure/monitors`, `GET /infrastructure/monitors/{id}`, `PUT /infrastructure/monitors/{id}`, `DELETE /infrastructure/monitors/{id}` |
|
||||
|
||||
**Creating Monitors via API**:
|
||||
```bash
|
||||
# Create a TCP port monitor for Redis
|
||||
curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"type": "port",
|
||||
"name": "Redis Shared - Port Check",
|
||||
"hostname": "redis-shared",
|
||||
"port": 6379,
|
||||
"interval": 60,
|
||||
"retryInterval": 60,
|
||||
"maxretries": 3,
|
||||
"notificationIDList": [],
|
||||
"accepted_statuscodes": ["200-299"]
|
||||
}'
|
||||
|
||||
# Create a PostgreSQL database monitor (note URL-encoded password)
|
||||
curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"type": "postgres",
|
||||
"name": "PostgreSQL Shared",
|
||||
"interval": 60,
|
||||
"retryInterval": 60,
|
||||
"maxretries": 3,
|
||||
"notificationIDList": [],
|
||||
"accepted_statuscodes": ["200-299"],
|
||||
"databaseConnectionString": "postgres://user:password@postgres-shared:5432/postgres"
|
||||
}'
|
||||
```
|
||||
|
||||
**Important**: When creating database monitors with passwords containing special characters (`/`, `=`, `+`, etc.), URL-encode them in the connection string (e.g., `/` → `%2F`, `=` → `%3D`).
|
||||
|
||||
---
|
||||
|
||||
### Jellyfin
|
||||
|
||||
Jellyfin is a GPU-accelerated media server that organizes, streams, and transcodes video, music, and photo libraries with hardware encoding via NVIDIA NVENC, enabling smooth 4K playback across multiple simultaneous clients without taxing the CPU. It provides a Netflix-like interface accessible through web browsers, mobile apps, and smart TV clients, with automatic metadata fetching, subtitle support, and user management for family sharing. The service stores configuration and cache on the SSD for responsive browsing while accessing massive media libraries on the 3.7TB HDD, supporting direct play when possible and GPU-accelerated transcoding when format conversion is needed.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `jellyfin/jellyfin:latest` |
|
||||
| **Container Name** | `jellyfin` |
|
||||
| **Access URL** | http://192.168.86.149:8096, https://media.schweitz.net |
|
||||
| **External Access** | LAN + Internet (via NPM reverse proxy) |
|
||||
| **Port Mapping** | 8096:8096 (HTTP), 8920:8920 (HTTPS), 7359:7359/udp (Discovery), 1900:1900/udp (DLNA) |
|
||||
| **Network Mode** | Bridge |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/jellyfin/config:/config`, `~/docker-data/jellyfin/cache:/cache`, `/mnt/media/jellyfin:/media:ro` |
|
||||
| **Environment** | `NVIDIA_VISIBLE_DEVICES=all`, `NVIDIA_DRIVER_CAPABILITIES=all` |
|
||||
| **User** | `1000:1000` (UID:GID) |
|
||||
| **Resource Limits** | Memory: 4GB, GPU: 11GB VRAM (shared) |
|
||||
| **GPU Required** | Yes (NVIDIA RTX 2080 Ti) |
|
||||
| **GPU Configuration** | NVENC hardware encoding, NVDEC hardware decoding |
|
||||
| **Dependencies** | NVIDIA Container Toolkit, NPM (for external access) |
|
||||
| **Media Storage** | /mnt/media/jellyfin (HDD) |
|
||||
| **Transcoding** | Hardware-accelerated (H.264/H.265) |
|
||||
|
||||
---
|
||||
|
||||
### Nextcloud
|
||||
|
||||
Nextcloud is a self-hosted cloud storage and collaboration platform providing file sync, sharing, calendar, contacts, and collaborative document editing with a web interface and mobile apps, replacing cloud services like Dropbox or Google Drive while maintaining full data sovereignty. It runs as a multi-container stack with a MariaDB database for metadata, Redis for caching and file locking, and the main PHP application container, with the application configuration stored on SSD for responsiveness while user data resides on the HDD for capacity. The service integrates behind Nginx Proxy Manager with SSL at https://cloud.schweitz.net, offering external access for file synchronization from anywhere while maintaining automated background job execution through the maintenance container's cron system.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `nextcloud:stable` |
|
||||
| **Container Name** | `nextcloud` |
|
||||
| **Access URL (LAN)** | http://192.168.86.149:8082 |
|
||||
| **Access URL (Public)** | https://cloud.schweitz.net |
|
||||
| **External Access** | Yes (via NPM reverse proxy with SSL) |
|
||||
| **Port Mapping** | 8082:80 (HTTP) |
|
||||
| **Network Mode** | Bridge (custom network: nextcloud_nextcloud-network) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/nextcloud/config:/var/www/html` (SSD), `/mnt/media/nextcloud/data:/var/www/html/data` (HDD) |
|
||||
| **Environment** | `MYSQL_HOST=nextcloud-db`, `MYSQL_DATABASE=nextcloud`, `MYSQL_USER=nextcloud`, `REDIS_HOST=nextcloud-redis`, `TZ=Europe/Amsterdam` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | MariaDB 10.11 (nextcloud-db), Redis Alpine (nextcloud-redis), NPM (reverse proxy), Maintenance container (cron jobs) |
|
||||
| **Database** | MariaDB on SSD (~100MB) |
|
||||
| **Cron Jobs** | Background tasks every 5 minutes (via maintenance container) |
|
||||
| **Storage Split** | Config/apps on SSD, user data on HDD |
|
||||
| **Features** | File sync, calendar, contacts, document editing, photo gallery, mobile apps |
|
||||
|
||||
---
|
||||
|
||||
### Samba
|
||||
|
||||
Samba provides SMB/CIFS network file sharing for seamless access to homelab storage from Windows, macOS, Linux, and mobile devices, exposing curated shares for media libraries, downloads, and backups with configurable read-only and read-write permissions. It runs as a single container on the samba_default network, serving three shares: Media (read-write access to Jellyfin content), Downloads (read-write for torrent clients), and Backups (read-only for safe data recovery). The service uses password authentication for the user jpmschweitzer and stores its minimal configuration on the SSD while directly mounting HDD paths for zero-copy file access with native performance.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `dperson/samba` |
|
||||
| **Container Name** | `samba` |
|
||||
| **Access URL** | \\\\192.168.86.149 or \\\\tower-of-joy |
|
||||
| **External Access** | LAN only (ports firewalled) |
|
||||
| **Port Mapping** | 139:139 (NetBIOS), 445:445 (SMB) |
|
||||
| **Network Mode** | Bridge (custom network: samba_default) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/samba:/share/config` (SSD config), `/mnt/media/jellyfin:/share/media` (Media share), `/mnt/media/downloads:/share/downloads` (Downloads share), `/mnt/media/backups:/share/backups:ro` (Backups read-only) |
|
||||
| **Environment** | `TZ=Europe/Amsterdam`, `USERID=1000`, `GROUPID=1000` |
|
||||
| **Command** | Share configs: Media (browseable, guest access), Downloads (no guest), Backups (read-only, browseable) |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | UFW firewall rules (ports 139, 445), Host Samba service disabled |
|
||||
| **Shares** | 3 total: Media (R/W), Downloads (R/W), Backups (R/O) |
|
||||
| **Authentication** | Username/password (jpmschweitzer) |
|
||||
| **Client Compatibility** | Windows, macOS, Linux, iOS, Android |
|
||||
|
||||
---
|
||||
|
||||
### Gitea
|
||||
|
||||
Gitea is a lightweight, self-hosted Git service providing repository hosting, issue tracking, pull requests, code review, and CI/CD integration through a clean web interface accessible via both HTTPS and SSH. It runs as a multi-container stack with a PostgreSQL database for metadata storage, offering GitHub-like functionality including organizations, teams, wikis, webhooks, and automated Actions workflows while maintaining complete data sovereignty and minimal resource overhead. The service stores all Git repositories and configuration on the SSD for fast access, integrates behind Nginx Proxy Manager with SSL at https://git.schweitz.net for web access, and exposes SSH on port 2222 for standard Git operations without conflicting with the host's SSH service.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Image** | `gitea/gitea:latest` |
|
||||
| **Container Name** | `gitea` |
|
||||
| **Access URL (LAN)** | http://192.168.86.149:3002 |
|
||||
| **Access URL (Public)** | https://git.schweitz.net |
|
||||
| **External Access** | Yes (via NPM reverse proxy with SSL) |
|
||||
| **Port Mapping** | 3002:3000 (HTTP), 2222:22 (SSH) |
|
||||
| **Network Mode** | Bridge (custom network: gitea_gitea-network) |
|
||||
| **Restart Policy** | `unless-stopped` |
|
||||
| **Volume Mounts** | `~/docker-data/gitea/data:/data` (SSD - repos, config) |
|
||||
| **Environment** | `USER_UID=1000`, `USER_GID=1000`, `GITEA__database__*` (PostgreSQL connection), `TZ=Europe/Amsterdam` |
|
||||
| **Resource Limits** | None |
|
||||
| **GPU Required** | No |
|
||||
| **Dependencies** | PostgreSQL 14 (gitea-db), NPM (reverse proxy) |
|
||||
| **Database** | PostgreSQL on SSD (~50MB) |
|
||||
| **SSH Access** | Port 2222 - `git clone ssh://git@git.schweitz.net:2222/user/repo.git` |
|
||||
| **Features** | Git hosting, Organizations/teams, Issues/PRs, Code review, Wikis, Webhooks, Gitea Actions (CI/CD), GitHub/GitLab migration |
|
||||
| **SSH Config Tip** | Add to `~/.ssh/config`: `Host git.schweitz.net` / `Port 2222` / `User git` for seamless cloning |
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference Tables
|
||||
|
||||
### Service Access Matrix
|
||||
|
||||
| Service | LAN URL | Internet Access | Primary Function |
|
||||
|---------|---------|-----------------|------------------|
|
||||
| **Portainer** | http://192.168.86.149:8001 | No | Container management |
|
||||
| **PostgreSQL Shared** | postgres-shared:5432 | No (internal) | Shared database server |
|
||||
| **Redis Shared** | redis-shared:6379 | No (internal) | Shared cache/session store |
|
||||
| **NPM** | http://192.168.86.149:81 | Yes (admin) | Reverse proxy admin |
|
||||
| **Code-Server** | https://code.schweitz.net | Yes | Browser-based IDE |
|
||||
| **Ollama** | http://192.168.86.149:11434 | No | ML model API |
|
||||
| **Headscale** | http://192.168.86.149:8085 | No | VPN control server |
|
||||
| **Uptime Kuma** | http://192.168.86.149:3001 | No | Uptime monitoring |
|
||||
| **Netdata** | http://192.168.86.149:19999 | No | System metrics |
|
||||
| **Heimdall** | http://192.168.86.149:8888 | No | Service dashboard |
|
||||
| **Organizr** | https://home.schweitz.net | Yes | Unified dashboard |
|
||||
| **Open WebUI** | http://192.168.86.149:82 | No | LLM chat interface |
|
||||
| **Core API** | http://192.168.86.149:8083 | No | API functions & infrastructure mgmt |
|
||||
| **Jellyfin** | https://media.schweitz.net | Yes | Media streaming |
|
||||
| **Nextcloud** | https://cloud.schweitz.net | Yes | Cloud storage & sync |
|
||||
| **Gitea** | https://git.schweitz.net | Yes | Git repository hosting |
|
||||
| **Samba** | \\\\192.168.86.149 | No | Network file shares |
|
||||
| **Watchtower** | N/A (background) | N/A | Auto-updates |
|
||||
| **Maintenance** | N/A (background) | N/A | Automated tasks |
|
||||
|
||||
---
|
||||
|
||||
### GPU-Enabled Services
|
||||
|
||||
| Service | GPU Usage | VRAM Requirements | Purpose |
|
||||
|---------|-----------|-------------------|---------|
|
||||
| **Ollama** | Compute, Utility | 2-10GB (model dependent) | LLM inference |
|
||||
| **Jellyfin** | Video Encode/Decode | ~1-2GB (during transcode) | Media transcoding |
|
||||
|
||||
**Total VRAM Available:** 11GB (RTX 2080 Ti)
|
||||
|
||||
---
|
||||
|
||||
### Storage Distribution
|
||||
|
||||
| Service | Config Location (SSD) | Data Location (HDD) | Typical Size |
|
||||
|---------|----------------------|---------------------|--------------|
|
||||
| **Portainer** | Docker volume: `portainer_data` | N/A | ~100MB |
|
||||
| **PostgreSQL Shared** | N/A | `~/docker-data/postgres-shared/data/` | Data: 100MB-5GB (depends on databases), Backups: variable |
|
||||
| **Redis Shared** | N/A | `~/docker-data/redis-shared/data/` | ~10-100MB (AOF + RDB snapshots) |
|
||||
| **NPM** | `~/docker-data/nginx-proxy-manager/` | N/A | ~50MB |
|
||||
| **Code-Server** | `~/.config/code-server/`, `~/docker-data/code-server/` | N/A | Config: ~5MB, Extensions: ~50-200MB, User data: ~50MB |
|
||||
| **Ollama** | `~/docker-data/ollama/models/` | Alt: `/mnt/media/ollama/` | 2-15GB per model |
|
||||
| **Headscale** | `~/docker-data/headscale/` | N/A | ~10MB |
|
||||
| **Uptime Kuma** | `~/docker-data/uptime-kuma/` | N/A | ~50MB |
|
||||
| **Netdata** | RAM-based (ephemeral) | N/A | ~200MB RAM |
|
||||
| **Heimdall** | `~/docker-data/heimdall/` | N/A | ~20MB |
|
||||
| **Organizr** | `~/docker-data/organizr/` | N/A | ~50MB |
|
||||
| **Open WebUI** | `~/docker-data/open-webui/` | N/A | ~100MB |
|
||||
| **Core API** | `~/docker-data/core-api/`, `/home/jpmschweitzer/Projects/portainer-core/services/core-api/` (source) | N/A | Venv: ~200MB, Logs: ~10MB |
|
||||
| **Jellyfin** | `~/docker-data/jellyfin/` | `/mnt/media/jellyfin/` | Config: ~500MB, Media: ~2TB |
|
||||
| **Nextcloud** | `~/docker-data/nextcloud/` | `/mnt/media/nextcloud/data/` | Config: ~200MB, DB: ~100MB, User data: variable |
|
||||
| **Gitea** | `~/docker-data/gitea/` | N/A | Data: ~100MB, DB: ~50MB, Repos: variable |
|
||||
| **Samba** | `~/docker-data/samba/` | Mounts: `/mnt/media/` (shares) | Config: ~5MB |
|
||||
| **Maintenance** | N/A | `/mnt/media/backups/` | ~94MB per backup |
|
||||
|
||||
**SSD Usage (docker-data):** ~6-11GB (configs, caches, databases)
|
||||
**HDD Usage (/mnt/media):** ~2.1TB / 3.6TB (58% used)
|
||||
|
||||
---
|
||||
|
||||
### Network Architecture
|
||||
|
||||
**As of 2025-11-15**, all services have been consolidated to the `docker-dataplane` bridge network for simplified service discovery and inter-container communication. This consolidation replaced 12+ legacy networks with a single unified network, enabling all services to communicate using container names as DNS hostnames.
|
||||
|
||||
| Network Name | Containers | Purpose |
|
||||
|--------------|------------|---------|
|
||||
| **docker-dataplane** | Ollama, Open WebUI, Core API, Qdrant, Uptime Kuma, PostgreSQL Shared, Redis Shared, Headscale, Nextcloud, Gitea, Samba, Watchtower, Maintenance, Organizr, Netdata | Unified service mesh for all containerized applications |
|
||||
| **host** | Portainer, NPM | Direct host port access for infrastructure management |
|
||||
|
||||
**Benefits of Consolidation**:
|
||||
- **Service Discovery**: All services reachable via `http://container-name:port` (e.g., `http://postgres-shared:5432`)
|
||||
- **Simplified Monitoring**: Uptime Kuma can monitor all services on docker-dataplane
|
||||
- **Shared Infrastructure**: postgres-shared and redis-shared accessible to all applications
|
||||
- **Network Cleanup**: Removed 7 obsolete networks (stacks_default, ai-dataplane, various stack-specific networks)
|
||||
|
||||
**Container Name Resolution Examples**:
|
||||
```bash
|
||||
# From any container on docker-dataplane
|
||||
curl http://uptime-kuma:3001 # Uptime Kuma API
|
||||
curl http://ollama:11434 # Ollama LLM API
|
||||
psql -h postgres-shared -U postgres # PostgreSQL connection
|
||||
redis-cli -h redis-shared # Redis connection
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Restart Policies
|
||||
|
||||
| Policy | Containers | Behavior |
|
||||
|--------|------------|----------|
|
||||
| **always** | Portainer | Restart on failure, on Docker daemon restart |
|
||||
| **unless-stopped** | All others | Restart on failure, but not after manual stop |
|
||||
|
||||
---
|
||||
|
||||
## Maintenance Schedule
|
||||
|
||||
| Service | Task | Frequency | Time |
|
||||
|---------|------|-----------|------|
|
||||
| **Watchtower** | Container updates | Daily | 4:00 AM |
|
||||
| **Maintenance** | Config backups | Daily | 3:00 AM |
|
||||
| **Maintenance** | Nextcloud background jobs | Every 5 minutes | Continuous |
|
||||
| **Maintenance** | Log cleanup | Weekly | Sunday 3:30 AM |
|
||||
| **Docker** | Image pruning | Monthly | 1st of month |
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: 2025-11-16*
|
||||
*System: tower-of-joy (tower-of-joy v0.5.0-optimization)*
|
||||
@@ -0,0 +1,266 @@
|
||||
# SYSTEM.md
|
||||
|
||||
**System documentation for LLM coding agents** - This file describes the computer system where this project resides, including hardware, OS, installed software, and environment details.
|
||||
|
||||
> Last updated: 2025-11-11
|
||||
|
||||
## System Overview
|
||||
|
||||
- **Hostname**: tower-of-joy
|
||||
- **User**: jpmschweitzer
|
||||
- **Home Directory**: /home/jpmschweitzer
|
||||
- **Project Location**: /home/jpmschweitzer/Projects/portainer-core
|
||||
|
||||
## Operating System
|
||||
|
||||
### Distribution
|
||||
- **OS**: Zorin OS 16.3
|
||||
- **Based on**: Ubuntu 20.04 (Focal Fossa)
|
||||
- **Kernel**: Linux 5.4.0-216-generic
|
||||
- **Architecture**: x86_64 (64-bit)
|
||||
|
||||
### Desktop Environment
|
||||
- **Display Server**: X11 (GDM)
|
||||
- **Desktop**: GNOME Shell (Zorin session mode)
|
||||
- **Session Manager**: gnome-session
|
||||
|
||||
### Locale & Timezone
|
||||
- **Language**: en_US.UTF-8
|
||||
- **Numeric/Time Format**: nl_NL.UTF-8
|
||||
- **Timezone**: Europe/Amsterdam (CET, +0100)
|
||||
|
||||
## Hardware Specifications
|
||||
|
||||
### CPU
|
||||
- **Model**: Intel Core i7-6700 @ 3.40GHz (6th Gen Skylake)
|
||||
- **Cores**: 4 physical cores, 8 threads (2 threads per core)
|
||||
- **Architecture**: x86_64
|
||||
- **Frequency**: 800 MHz - 4000 MHz (currently ~3666 MHz)
|
||||
- **Cache**:
|
||||
- L1d: 128 KiB
|
||||
- L1i: 128 KiB
|
||||
- L2: 1 MiB
|
||||
- L3: 8 MiB
|
||||
- **Virtualization**: VT-x supported
|
||||
- **Notable Flags**: AVX, AVX2, AES-NI, SSE4.1, SSE4.2, FMA
|
||||
|
||||
### Memory
|
||||
- **Total RAM**: 16 GiB
|
||||
- **Available**: ~9.6 GiB (typical)
|
||||
- **Swap**: 2.0 GiB
|
||||
|
||||
### Storage
|
||||
|
||||
**System has 2 disks with total capacity of 4.2TB:**
|
||||
|
||||
#### Disk 1: System SSD (/dev/sda)
|
||||
- **Model**: Crucial CT525MX300SSD1 (525GB SSD)
|
||||
- **Partition**: /dev/sda1
|
||||
- **Filesystem**: ext4
|
||||
- **Total Size**: 489 GB
|
||||
- **Used**: 92 GB (21%)
|
||||
- **Available**: 365 GB
|
||||
- **Mount Point**: `/` (root)
|
||||
- **Purpose**: Operating system, Docker containers, application data
|
||||
|
||||
#### Disk 2: Media HDD (/dev/sdb)
|
||||
- **Model**: Seagate IronWolf NE ST4000NE001 (4TB NAS-grade HDD)
|
||||
- **Total Size**: 3.7 TB
|
||||
- **Filesystem**: ext4
|
||||
- **Label**: "media"
|
||||
- **UUID**: f4300e91-3f51-45a0-b038-03335c5bd792
|
||||
- **Mount Status**: ⚠️ **Currently unmounted** (not in /etc/fstab)
|
||||
- **Purpose**: Media storage for Jellyfin, Nextcloud data, backups
|
||||
- **Drive Type**: NAS-optimized (24/7 operation, multi-user workloads)
|
||||
|
||||
**Total Storage Capacity**: 4.2 TB
|
||||
|
||||
### Graphics
|
||||
- **GPU**: NVIDIA GeForce RTX 2080 Ti (TU102, Rev. A)
|
||||
- **VRAM**: 11 GB (11018 MiB)
|
||||
- **Driver**: NVIDIA 470.256.02
|
||||
- **CUDA Version**: 11.4
|
||||
- **Bus**: PCIe 0a:00.0
|
||||
- **Current Usage**: ~390 MiB VRAM (mostly X11/GNOME)
|
||||
- **Power**: 260W TDP
|
||||
|
||||
**Note**: NVCC (CUDA compiler) is not currently in PATH, but CUDA drivers are installed.
|
||||
|
||||
## Development Tools & Languages
|
||||
|
||||
### Programming Languages
|
||||
|
||||
#### Python
|
||||
- **Version**: 3.8.10 (system default)
|
||||
- **pip**: 25.3 (Python 3.10 in user site-packages)
|
||||
- **Location**: /usr/bin/python3
|
||||
- **Python 2.x**: Not installed
|
||||
- **Virtual Environments**:
|
||||
- virtualenv: Not installed
|
||||
- Conda: Not installed
|
||||
- venv module: Available (built-in)
|
||||
|
||||
#### Node.js & JavaScript
|
||||
- **Node.js**: v24.11.0
|
||||
- **npm**: 11.6.1
|
||||
- **Version Manager**: NVM installed at /home/jpmschweitzer/.nvm
|
||||
|
||||
#### Java
|
||||
- **Version**: Java 21.0.4 LTS (Oracle JDK)
|
||||
- **Runtime**: Java(TM) SE Runtime Environment (build 21.0.4+8-LTS-274)
|
||||
- **VM**: Java HotSpot 64-Bit Server VM
|
||||
|
||||
#### C/C++
|
||||
- **GCC**: 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)
|
||||
- **Make**: GNU Make 4.2.1
|
||||
- **CMake**: Not installed
|
||||
|
||||
#### Other Languages
|
||||
- **Go**: Not installed
|
||||
- **Rust**: Not installed
|
||||
|
||||
### Version Control
|
||||
- **Git**: 2.25.1
|
||||
|
||||
### Containerization & Virtualization
|
||||
- **Docker**: 28.1.1, build 4eba377
|
||||
|
||||
### Editors & IDEs
|
||||
- **Vim**: 8.1 (2018 May 18)
|
||||
- **VS Code**: Not installed
|
||||
|
||||
### Command Line Tools
|
||||
- **Shell**: Bash 5.0.17
|
||||
- **curl**: 7.68.0
|
||||
- **wget**: 1.20.3
|
||||
- **SSH**: OpenSSH 8.2p1 Ubuntu-4ubuntu0.13
|
||||
|
||||
## GPU & CUDA Information
|
||||
|
||||
### NVIDIA GPU Details
|
||||
The system has an NVIDIA RTX 2080 Ti with CUDA support, suitable for:
|
||||
- Machine learning and deep learning workloads
|
||||
- CUDA-accelerated computing
|
||||
- GPU rendering and compute tasks
|
||||
- Parallel processing
|
||||
|
||||
### CUDA Configuration
|
||||
- **Driver Version**: 470.256.02
|
||||
- **CUDA Toolkit Version**: 11.4 (driver supports)
|
||||
- **Compute Capability**: 7.5 (Turing architecture)
|
||||
- **NVCC**: Not in PATH (may need manual setup)
|
||||
|
||||
### GPU Usage Considerations
|
||||
When working with GPU-accelerated code:
|
||||
- Ensure CUDA toolkit is properly installed if needed
|
||||
- Use appropriate CUDA version compatibility (11.4 or compatible)
|
||||
- PyTorch/TensorFlow should use CUDA 11.x compatible builds
|
||||
- Monitor VRAM usage (11 GB total, ~10.6 GB available for compute)
|
||||
|
||||
## System Capabilities & Recommendations
|
||||
|
||||
### Suitable For
|
||||
- **Web Development**: Node.js, npm available
|
||||
- **Python Development**: Python 3.8 with pip
|
||||
- **Java Development**: Java 21 LTS
|
||||
- **Machine Learning**: CUDA-capable GPU with 11GB VRAM
|
||||
- **Containerized Development**: Docker available
|
||||
- **Compiled Languages**: GCC toolchain available
|
||||
- **Media Server**: 3.7TB NAS-grade storage for Jellyfin/Plex
|
||||
- **NAS/File Server**: Seagate IronWolf drive optimized for 24/7 operation
|
||||
- **Cloud Storage**: Ample space for Nextcloud deployments
|
||||
- **Home Server**: Suitable for comprehensive home lab setup
|
||||
|
||||
### Limitations
|
||||
- No Rust toolchain (needs installation)
|
||||
- No Go compiler (needs installation)
|
||||
- CMake not installed (needed for some C/C++ projects)
|
||||
- VS Code not installed (Vim available as alternative)
|
||||
- CUDA compiler not in PATH
|
||||
|
||||
### Environment Notes
|
||||
- NVM is available for Node.js version management
|
||||
- Python 3.8 is the system default (older, consider pyenv for newer versions)
|
||||
- pip is installed in user site-packages (Python 3.10 version)
|
||||
- Docker is available for containerized workflows
|
||||
|
||||
## Package Management
|
||||
|
||||
### System Package Manager
|
||||
- **APT**: Available (Ubuntu/Debian package manager)
|
||||
- Use `sudo apt install <package>` for system packages
|
||||
|
||||
### Language-Specific Package Managers
|
||||
- **Python**: pip3 (25.3)
|
||||
- **Node.js**: npm (11.6.1), managed via NVM
|
||||
- **Java**: Maven/Gradle likely needed (not verified)
|
||||
|
||||
## Network Information
|
||||
- SSH client available (OpenSSH 8.2p1)
|
||||
- Standard network tools available (curl, wget)
|
||||
|
||||
## Usage Notes for LLM Agents
|
||||
|
||||
### Before Installing New Software
|
||||
1. Check if the tool is already installed using `which <command>`
|
||||
2. Check available disk space:
|
||||
- System SSD: 365 GB available (for OS and containers)
|
||||
- Media HDD: 3.7 TB available (currently unmounted - needs mounting)
|
||||
3. Use appropriate package manager (apt, pip, npm, etc.)
|
||||
4. Consider using Docker for isolated environments
|
||||
5. **Mount the 4TB media drive** before deploying data-intensive services:
|
||||
- Recommended mount point: `/mnt/media` or `/media/storage`
|
||||
- Add to `/etc/fstab` for automatic mounting on boot
|
||||
- UUID: `f4300e91-3f51-45a0-b038-03335c5bd792`
|
||||
|
||||
### GPU Development
|
||||
1. Verify CUDA toolkit path if developing GPU code
|
||||
2. Check GPU memory availability with `nvidia-smi`
|
||||
3. Use CUDA 11.x compatible libraries
|
||||
4. Monitor GPU utilization to avoid OOM errors
|
||||
|
||||
### Python Development
|
||||
1. System Python is 3.8.10 (older version)
|
||||
2. Consider using venv for project isolation
|
||||
3. pip is available but points to Python 3.10 libs in user space
|
||||
4. May need to install python3-venv: `sudo apt install python3-venv`
|
||||
|
||||
### Node.js Development
|
||||
1. NVM is installed for version management
|
||||
2. Current Node.js is v24.11.0 (latest as of 2024)
|
||||
3. npm 11.6.1 is available
|
||||
|
||||
### Docker Usage
|
||||
1. Docker 28.1.1 is installed
|
||||
2. Useful for consistent development environments
|
||||
3. Can isolate dependencies and avoid system conflicts
|
||||
|
||||
### Storage Management (Dual-Disk Setup)
|
||||
1. **System SSD (/dev/sda)**: Use for:
|
||||
- Operating system
|
||||
- Docker images and container configs
|
||||
- Application databases (small, performance-critical)
|
||||
- Cache directories
|
||||
|
||||
2. **Media HDD (/dev/sdb)**: Use for:
|
||||
- Jellyfin/Plex media libraries
|
||||
- Nextcloud user data
|
||||
- Backups and archives
|
||||
- Large file storage
|
||||
- Any data-intensive workloads
|
||||
|
||||
3. **Best Practices**:
|
||||
- Keep Docker container configs on SSD for performance
|
||||
- Store media files on HDD (they're sequential access, HDD is fine)
|
||||
- Use bind mounts to map HDD storage into containers
|
||||
- Example: `-v /mnt/media/jellyfin:/media:ro` in Docker
|
||||
|
||||
4. **Before First Use**:
|
||||
- Mount the media drive (see step 5 in "Before Installing New Software")
|
||||
- Verify mount with `df -h /mnt/media`
|
||||
- Set appropriate permissions: `sudo chown -R $USER:$USER /mnt/media`
|
||||
|
||||
---
|
||||
|
||||
*Generated automatically on 2025-11-11, updated with storage configuration*
|
||||
*For project-specific guidelines, see [AGENTS.md](./AGENTS.md)*
|
||||
@@ -0,0 +1,200 @@
|
||||
# Maintenance Scripts Reference
|
||||
|
||||
Shell scripts for common maintenance tasks located in `/scripts/`.
|
||||
|
||||
## Available Scripts
|
||||
|
||||
| Script | Description | Usage |
|
||||
|--------|-------------|-------|
|
||||
| `gpu-check.sh` | Verify GPU passthrough in containers | `./scripts/gpu-check.sh` |
|
||||
| `health-check.sh` | Check all services and report status | `./scripts/health-check.sh` |
|
||||
| `setup-kuma-monitors.sh` | Manual guide for configuring Uptime Kuma monitors | `./scripts/setup-kuma-monitors.sh` |
|
||||
| `setup-kuma-monitors.py` | **Automated** Uptime Kuma monitor setup via API | `source .venv/bin/activate && python3 scripts/setup-kuma-monitors.py` |
|
||||
| `backup-configs.sh` | Backup all Docker configs | `./scripts/backup-configs.sh` |
|
||||
| `disk-usage.sh` | Report disk usage for SSD and HDD | `./scripts/disk-usage.sh` |
|
||||
| `update-stacks.sh` | Pull latest images and update containers | `./scripts/update-stacks.sh <stack-name>` |
|
||||
| `cleanup.sh` | Clean up unused Docker resources | `./scripts/cleanup.sh` |
|
||||
|
||||
## Making Scripts Executable
|
||||
|
||||
```bash
|
||||
# Make all scripts executable
|
||||
chmod +x scripts/*.sh
|
||||
|
||||
# Or individually
|
||||
chmod +x scripts/health-check.sh
|
||||
```
|
||||
|
||||
## Scheduling with Cron
|
||||
|
||||
Add to crontab for automated maintenance:
|
||||
|
||||
```bash
|
||||
# Edit crontab
|
||||
crontab -e
|
||||
|
||||
# Examples:
|
||||
# Daily health check at 8 AM
|
||||
0 8 * * * /home/jpmschweitzer/Projects/portainer-core/scripts/health-check.sh >> /var/log/portainer-core-health.log 2>&1
|
||||
|
||||
# Weekly cleanup on Sunday at 3 AM
|
||||
0 3 * * 0 /home/jpmschweitzer/Projects/portainer-core/scripts/cleanup.sh
|
||||
|
||||
# Daily backup at 2 AM
|
||||
0 2 * * * /home/jpmschweitzer/Projects/portainer-core/scripts/backup-configs.sh
|
||||
```
|
||||
|
||||
## Script Details
|
||||
|
||||
### GPU Check (`gpu-check.sh`)
|
||||
|
||||
Verifies GPU passthrough is working in GPU-enabled containers (Ollama, Jellyfin).
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/gpu-check.sh
|
||||
```
|
||||
|
||||
**Output:**
|
||||
- Lists all running containers with GPU access
|
||||
- Runs `nvidia-smi` inside each container
|
||||
- Reports any containers that fail GPU detection
|
||||
|
||||
### Health Check (`health-check.sh`)
|
||||
|
||||
Checks status of all deployed services and generates a health report.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/health-check.sh
|
||||
```
|
||||
|
||||
**Checks:**
|
||||
- Container running status
|
||||
- Container health status (if health check defined)
|
||||
- Port accessibility
|
||||
- Basic connectivity tests
|
||||
|
||||
### Uptime Kuma Monitor Setup
|
||||
|
||||
Two versions available:
|
||||
|
||||
**Manual Script (`setup-kuma-monitors.sh`):**
|
||||
- Interactive guide for adding monitors
|
||||
- Shows recommended settings for each service
|
||||
- Good for understanding monitor configuration
|
||||
|
||||
**Automated Script (`setup-kuma-monitors.py`):**
|
||||
- Python script using Uptime Kuma API
|
||||
- Automatically creates monitors for all services
|
||||
- Requires Uptime Kuma API key
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Automated setup
|
||||
source .venv/bin/activate
|
||||
python3 scripts/setup-kuma-monitors.py
|
||||
```
|
||||
|
||||
### Backup Configs (`backup-configs.sh`)
|
||||
|
||||
Backs up Docker container configurations and important data.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/backup-configs.sh
|
||||
```
|
||||
|
||||
**What it backs up:**
|
||||
- Docker Compose files from `/stacks/`
|
||||
- Container configs from `/home/jpmschweitzer/docker-data/`
|
||||
- Project documentation
|
||||
- Excludes large media files (those are backed up separately)
|
||||
|
||||
**Backup location:**
|
||||
- `/mnt/media/backups/portainer-core/`
|
||||
|
||||
See [Backup Procedures](../guides/backup-procedures.md) for comprehensive backup strategy.
|
||||
|
||||
### Disk Usage (`disk-usage.sh`)
|
||||
|
||||
Reports disk usage breakdown for SSD and HDD storage.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/disk-usage.sh
|
||||
```
|
||||
|
||||
**Output:**
|
||||
- Total SSD usage (`/home/jpmschweitzer/docker-data/`)
|
||||
- Total HDD usage (`/mnt/media/`)
|
||||
- Per-service breakdown
|
||||
- Available space warnings
|
||||
|
||||
### Update Stacks (`update-stacks.sh`)
|
||||
|
||||
Pulls latest images and updates a specific stack.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/update-stacks.sh <stack-name>
|
||||
|
||||
# Examples:
|
||||
./scripts/update-stacks.sh jellyfin
|
||||
./scripts/update-stacks.sh core-api
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Pulls latest images for the stack
|
||||
2. Stops containers gracefully
|
||||
3. Recreates containers with new images
|
||||
4. Removes old images
|
||||
5. Verifies containers started successfully
|
||||
|
||||
**Note:** Watchtower handles this automatically for most services. Use this script for manual updates or services excluded from Watchtower.
|
||||
|
||||
### Cleanup (`cleanup.sh`)
|
||||
|
||||
Cleans up unused Docker resources to free disk space.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/cleanup.sh
|
||||
```
|
||||
|
||||
**What it removes:**
|
||||
- Stopped containers
|
||||
- Unused images
|
||||
- Dangling build cache
|
||||
- Unused volumes (with confirmation prompt)
|
||||
- Unused networks
|
||||
|
||||
**Warning:** Always review what will be removed before confirming volume deletion.
|
||||
|
||||
## Script Guidelines
|
||||
|
||||
All scripts follow these conventions:
|
||||
|
||||
- Include error handling and exit codes
|
||||
- Use absolute paths for reliability
|
||||
- Log output for debugging
|
||||
- Exit with status codes (0 = success, non-zero = failure)
|
||||
- Include help text with `-h` or `--help` flags
|
||||
- Non-destructive by default (ask before deleting)
|
||||
|
||||
## Creating New Scripts
|
||||
|
||||
When adding new maintenance scripts:
|
||||
|
||||
1. Place in `/scripts/` directory
|
||||
2. Use `.sh` extension for shell scripts
|
||||
3. Make executable: `chmod +x scripts/your-script.sh`
|
||||
4. Add to this documentation
|
||||
5. Include help text and error handling
|
||||
6. Test thoroughly before scheduling with cron
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Backup Procedures](../guides/backup-procedures.md) - Comprehensive backup strategy
|
||||
- [Stacks Reference](stacks.md) - Stack deployment and management
|
||||
- [Automation Reference](AUTOMATION.md) - Portainer REST API automation
|
||||
@@ -0,0 +1,185 @@
|
||||
# Docker Compose Stacks Reference
|
||||
|
||||
Complete reference for all Docker Compose stacks in the portainer-core infrastructure.
|
||||
|
||||
## Deployment
|
||||
|
||||
See the [core-api OpenAPI documentation](http://localhost:8083/docs) for infrastructure management REST endpoints.
|
||||
|
||||
All stacks are located in the `/stacks/` directory and version-controlled.
|
||||
|
||||
## Stack Inventory
|
||||
|
||||
### Phase 1: Foundation
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Portainer** | `portainer.yml` | 8080, 8443 | No | Container management UI |
|
||||
| **Nginx Proxy Manager** | `nginx-proxy-manager.yml` | 8000, 80, 443 | No | Reverse proxy and unified web interface |
|
||||
| **Ollama** | `ollama.yml` | 11434 | **Yes** | ML model serving with GPU acceleration |
|
||||
|
||||
### Phase 2: Networking
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Headscale** | `headscale.yml` | 8085, 9090 | No | Self-hosted Tailscale control server |
|
||||
|
||||
### Phase 3: Monitoring
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Uptime Kuma** | `uptime-kuma.yml` | 3001 | No | Service availability monitoring |
|
||||
| **Netdata** | `netdata.yml` | 19999 | No | Real-time system performance monitoring |
|
||||
| **Heimdall** | `heimdall.yml` | 8888, 8889 | No | Application dashboard |
|
||||
|
||||
### Phase 4: Optimization
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Watchtower** | `watchtower.yml` | - | No | Automatic container updates |
|
||||
| **Duplicati** | `duplicati.yml` | 8200 | No | Backup solution |
|
||||
|
||||
### Applications
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **Jellyfin** | `jellyfin.yml` | 8096, 8920, 7359, 1900 | **Yes** | Media server with GPU transcoding |
|
||||
| **Nextcloud** | `nextcloud.yml` | 8082 | No | Cloud storage (uses shared PostgreSQL and Redis) |
|
||||
| **Gitea** | `gitea.yml` | 3002, 2222 | No | Git repository hosting (includes PostgreSQL) |
|
||||
| **Samba** | `samba.yml` | 139, 445 | No | Network file sharing |
|
||||
| **Open WebUI** | `open-webui.yml` | 8081 | No | AI chat interface with Ollama integration |
|
||||
| **Core API** | `core-api.yml` | 8083 | No | Infrastructure management and AI orchestration |
|
||||
| **Qdrant** | `qdrant.yml` | 6333, 6334 | No | Vector database for embeddings |
|
||||
| **Organizr** | `organizr.yml` | 8084 | No | Unified dashboard |
|
||||
|
||||
### Shared Infrastructure
|
||||
|
||||
| Stack | File | Ports | GPU | Description |
|
||||
|-------|------|-------|-----|-------------|
|
||||
| **PostgreSQL Shared** | `postgres-shared.yml` | 5432 | No | Shared database for Nextcloud |
|
||||
| **Redis Shared** | `redis-shared.yml` | 6379 | No | Shared cache for Nextcloud |
|
||||
|
||||
## Port Allocation
|
||||
|
||||
### Infrastructure Services (8000-8099)
|
||||
- 8000: Nginx Proxy Manager (unified web interface)
|
||||
- 8080: Portainer
|
||||
- 8081: Open WebUI
|
||||
- 8082: Nextcloud
|
||||
- 8083: Core API
|
||||
- 8084: Organizr
|
||||
- 8085: Headscale
|
||||
- 8096: Jellyfin
|
||||
|
||||
### Git & Development Services
|
||||
- 2222: Gitea SSH
|
||||
- 3002: Gitea HTTP
|
||||
|
||||
### Monitoring Services (3000-3999, 19000-19999)
|
||||
- 3001: Uptime Kuma
|
||||
- 8200: Duplicati
|
||||
- 8888: Heimdall
|
||||
- 19999: Netdata
|
||||
|
||||
### ML/API Services (11000+)
|
||||
- 11434: Ollama
|
||||
- 6333: Qdrant HTTP
|
||||
- 6334: Qdrant gRPC
|
||||
|
||||
### Database Services
|
||||
- 5432: PostgreSQL (shared)
|
||||
- 6379: Redis (shared)
|
||||
|
||||
### Network Services
|
||||
- 80: HTTP (NPM reverse proxy)
|
||||
- 443: HTTPS (NPM reverse proxy)
|
||||
- 139, 445: Samba/SMB
|
||||
- 9090: Headscale metrics
|
||||
|
||||
## Storage Convention
|
||||
|
||||
All stacks follow the dual-disk strategy:
|
||||
|
||||
**SSD (Performance):**
|
||||
- Configs: `/home/jpmschweitzer/docker-data/<service>/config`
|
||||
- Cache: `/home/jpmschweitzer/docker-data/<service>/cache`
|
||||
- Databases: `/home/jpmschweitzer/docker-data/<service>/db`
|
||||
|
||||
**HDD (Capacity):**
|
||||
- User content: `/mnt/media/<service>/data`
|
||||
- Media files: `/mnt/media/<service>/media`
|
||||
- Backups: `/mnt/media/backups/<service>`
|
||||
|
||||
See [Shared Infrastructure Architecture](../architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md) for database and cache sharing details.
|
||||
|
||||
## GPU Services
|
||||
|
||||
Stacks requiring GPU access (marked with **Yes** above):
|
||||
- `ollama.yml` - ML model inference
|
||||
- `jellyfin.yml` - Hardware transcoding
|
||||
|
||||
**Prerequisites:**
|
||||
- NVIDIA Container Toolkit installed
|
||||
- GPU verified: `docker run --rm --gpus all nvidia/cuda:11.4.0-base-ubuntu20.04 nvidia-smi`
|
||||
|
||||
See [GPU Docker Configuration](../guides/gpu-docker-config.md) for setup details.
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
### Before Deploying
|
||||
|
||||
1. **Review environment variables** - Change default passwords!
|
||||
2. **Create directories** - Ensure volume paths exist
|
||||
3. **Check ports** - Verify no conflicts with existing services
|
||||
4. **GPU services** - Confirm NVIDIA toolkit installed
|
||||
5. **Update STATUS.md** - Plan the deployment
|
||||
|
||||
### After Deploying
|
||||
|
||||
1. **Test service** - Access web UI or API endpoint
|
||||
2. **Check logs** - `docker logs <container-name>`
|
||||
3. **Verify GPU** - `docker exec <container> nvidia-smi` (if applicable)
|
||||
4. **Update documentation** - Add to STATUS.md and CHANGELOG.md
|
||||
5. **Configure backup** - Add to Duplicati backup job
|
||||
6. **Add monitoring** - Configure Uptime Kuma checks
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Update a Stack
|
||||
|
||||
```bash
|
||||
# Pull latest images
|
||||
docker compose -f stacks/<stack-name>.yml pull
|
||||
|
||||
# Recreate containers with new images
|
||||
docker compose -f stacks/<stack-name>.yml up -d
|
||||
|
||||
# Or let Watchtower handle it automatically
|
||||
```
|
||||
|
||||
### Backup Stack Configuration
|
||||
|
||||
Stacks are version-controlled in the `/stacks/` directory. Backup container data separately using the backup procedures.
|
||||
|
||||
See [Backup Procedures](../guides/backup-procedures.md) for details.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
- Container won't start: `docker logs <container-name>`
|
||||
- Port conflicts: `sudo netstat -tulpn | grep <port>`
|
||||
- Permission issues: Check volume path ownership
|
||||
- GPU not detected: Verify NVIDIA toolkit and restart Docker
|
||||
|
||||
## Automation
|
||||
|
||||
The project includes automation scripts for stack management:
|
||||
|
||||
- `update-stack.sh` - Pull and update specific stack
|
||||
- See [Automation Reference](AUTOMATION.md) for Portainer REST API usage
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Container Reference](CONTAINERS.md) - Complete container profiles
|
||||
- [System Specifications](SYSTEM.md) - Hardware and software specs
|
||||
- [Shared Infrastructure Architecture](../architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md) - Database/cache sharing
|
||||
- [Maintenance Scripts](scripts.md) - Automated maintenance tasks
|
||||
Reference in New Issue
Block a user