# Container Platform Implementation Plan

> Implementation Date: 2025-11-11
> System: tower-of-joy
> Solution: Portainer + Docker Compose + Nginx Proxy Manager + Headscale + Ollama
> Estimated Total Time: 8-20 hours for base infrastructure (applications in backlog)

## Overview

This document provides a detailed, phase-by-phase implementation plan for setting up a home server using Portainer, Docker Compose, Nginx Proxy Manager (reverse proxy), Headscale (SDN), and Ollama (ML models). Each phase includes specific steps, commands, and **functionality tests** to verify successful completion before proceeding.

**Key Features:**
- **Web-based container management** (Portainer)
- **Unified reverse proxy interface** (Nginx Proxy Manager on port 8000)
- **GPU-accelerated ML model serving** (Ollama/vLLM with RTX 2080 Ti)
- **Dual-disk storage strategy** (489GB SSD + 3.7TB HDD)
- **Game server integration** (AMP compatibility)
- **Secure remote access** (Headscale/Tailscale SDN)

**Note:** This plan focuses on establishing the base infrastructure first. Application deployments (Jellyfin, Nextcloud, Samba) have been moved to the Phase Backlog section and will be implemented after the infrastructure is solid.

## Dual-Disk Storage Strategy

This system has **two disks with 4.2TB total capacity:**

- **SSD (489GB)**: System drive (`/dev/sda`) - OS, Docker images, container configs, databases
- **HDD (3.7TB)**: Media drive (`/dev/sdb`) - User content, media files, Nextcloud data, backups

**Phase 1** includes mounting the 4TB media drive to `/mnt/media` and configuring it for automatic mounting. All subsequent container deployments will use this dual-disk strategy:
- Fast SSD for configs and performance-critical data
- Large HDD for bulk storage and user content

## Pre-Implementation Checklist

Before starting, verify the following:

- [ ] System: tower-of-joy (Zorin OS 16.3)
- [ ] Docker version: 28.1.1 or later
- [ ] Available disk space: At least 50GB free
- [ ] NVIDIA GPU: RTX 2080 Ti detected
- [ ] Root/sudo access available
- [ ] Internet connection stable
- [ ] Backup of important data completed

### Pre-Flight System Check

```bash
# Verify Docker installation
docker --version
docker ps

# Check NVIDIA GPU
nvidia-smi

# Check available disk space
df -h /

# Check system resources
free -h
```

**Expected Results:**
- Docker version 28.1.1 or later
- NVIDIA driver showing RTX 2080 Ti
- At least 50GB free disk space
- At least 12GB RAM available

---

## Phase 1: Foundation Setup

**Goal:** Install NVIDIA Container Toolkit, deploy Portainer, configure dual-disk storage, set up reverse proxy, and prepare ML infrastructure
**Estimated Time:** 2-4 hours
**Prerequisites:** Docker installed, sudo access

**Phase Overview:**
1. Install NVIDIA Container Toolkit for GPU support
2. Deploy Portainer for container management
3. Configure GPU management in Portainer
4. Mount 4TB media drive for container user content
5. Configure AMP integration and game server storage
6. Deploy Nginx Proxy Manager for unified web interface
7. Prepare infrastructure for ML model containers (Ollama/vLLM)

### Step 1.1: Install NVIDIA Container Toolkit

The NVIDIA Container Toolkit enables GPU access within Docker containers, essential for Jellyfin hardware transcoding.

#### Commands

```bash
# Add NVIDIA Docker repository
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Update package list
sudo apt-get update

# Install NVIDIA Container Toolkit
sudo apt-get install -y nvidia-container-toolkit

# Configure Docker to use NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker

# Restart Docker daemon
sudo systemctl restart docker
```

#### Functionality Test 1.1: Verify GPU Access in Docker

```bash
# Test GPU access in a container
docker run --rm --gpus all nvidia/cuda:11.4.0-base-ubuntu20.04 nvidia-smi
```

**Expected Output:**
- NVIDIA-SMI output showing RTX 2080 Ti
- Driver version: 470.256.02
- CUDA Version: 11.4

**Troubleshooting:**
- If `--gpus` flag not recognized: Docker daemon didn't restart properly, retry `sudo systemctl restart docker`
- If GPU not visible: Check `sudo nvidia-ctk runtime configure --runtime=docker` output for errors
- If CUDA version mismatch: Acceptable as long as GPU is detected

✅ **Checkpoint 1.1:** GPU successfully accessible in Docker containers

---

### Step 1.2: Deploy Portainer

Portainer provides the web-based management interface for all our containers.

#### Commands

```bash
# Create Portainer volume for persistent data
docker volume create portainer_data

# Deploy Portainer CE (Community Edition)
docker run -d \
  -p 8080:9000 \
  -p 8443:9443 \
  --name=portainer \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  portainer/portainer-ce:latest
```

**Port Configuration:**
- Port 8080 (HTTP) - Main Portainer web UI
- Port 8443 (HTTPS) - Portainer HTTPS access
- Port 8000 - **Reserved for Nginx Proxy Manager** (see Step 1.6)

**Note:** This uses port 8080 for Portainer. AMP currently uses 8080 and will need to be reconfigured to port 8081 (see Step 1.5).

#### Functionality Test 1.2: Access Portainer Web UI

```bash
# Check Portainer is running
docker ps | grep portainer

# Check Portainer logs
docker logs portainer
```

**Access Portainer:**
1. Open browser and navigate to: `http://localhost:8080` or `http://tower-of-joy:8080`
2. First-time setup: Create admin account
   - Username: (choose your username)
   - Password: (minimum 12 characters, use strong password)
3. Click "Get Started"
4. Select "local" environment

**Expected Results:**
- Portainer login page loads
- Can create admin account
- Dashboard shows "local" Docker environment
- Can see existing Docker containers (including Portainer itself)

✅ **Checkpoint 1.2:** Portainer web UI accessible and showing local Docker environment

---

### Step 1.3: Configure Portainer for GPU Support

Enable GPU device selection in Portainer settings.

#### Steps

1. In Portainer web UI, navigate to: **Settings** → **Docker**
2. Scroll to **Available GPU Devices**
3. Click **Enable GPU Management**
4. Save settings

#### Functionality Test 1.3: Verify GPU Visibility in Portainer

1. Go to **Home** → **local** → **Containers**
2. Click **Add Container**
3. Scroll to **Runtime & Resources**
4. Check if **GPU** toggle is available

**Expected Results:**
- GPU toggle visible in container creation
- Can enable/disable GPU access per container

✅ **Checkpoint 1.3:** Portainer configured for GPU management

---

### Step 1.4: Mount the 4TB Media Drive

Mount the Seagate IronWolf 4TB drive for storing all container user content (media, Nextcloud data, game saves, etc.).

#### System Storage Strategy

**Two-Disk Configuration:**
- **SSD (/dev/sda)**: Container configs, databases, Docker images, OS
- **HDD (/dev/sdb)**: User content, media files, Nextcloud data, backups

#### Commands

```bash
# Create mount point
sudo mkdir -p /mnt/media

# Mount the media drive
sudo mount /dev/sdb /mnt/media

# Verify mount
df -h /mnt/media
```

**Expected Output:**
```
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        3.7T  XXG  X.XTT  XX% /mnt/media
```

#### Add to /etc/fstab for Automatic Mounting

```bash
# Backup current fstab
sudo cp /etc/fstab /etc/fstab.backup

# Add media drive to fstab
echo "UUID=f4300e91-3f51-45a0-b038-03335c5bd792 /mnt/media ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab

# Test fstab configuration
sudo mount -a

# Verify all mounts
df -h
```

#### Set Permissions

```bash
# Set ownership to current user
sudo chown -R $USER:$USER /mnt/media

# Verify permissions
ls -la /mnt/media
```

#### Create Directory Structure for Container Data

```bash
# Create base directories for container content
mkdir -p /mnt/media/jellyfin
mkdir -p /mnt/media/nextcloud
mkdir -p /mnt/media/backups
mkdir -p /mnt/media/game-servers
mkdir -p /mnt/media/downloads

# Verify structure
tree -L 1 /mnt/media
```

#### Functionality Test 1.4: Verify Media Drive Setup

```bash
# Check mount point
df -h /mnt/media

# Test write permissions
touch /mnt/media/test-file.txt
echo "Media drive test" > /mnt/media/test-file.txt
cat /mnt/media/test-file.txt
rm /mnt/media/test-file.txt

# Verify directory structure
ls -la /mnt/media
```

**Expected Results:**
- `/mnt/media` shows 3.7TB available
- Can create and write files
- Directory structure created successfully
- Owned by current user
- Drive will auto-mount on reboot (via fstab)

**Storage Usage Policy Going Forward:**
- ✅ **ALL user content** → `/mnt/media/*` (HDD)
- ✅ **Container configs** → `~/docker-data/*` (SSD)
- ✅ **Docker images** → `/var/lib/docker` (SSD)

✅ **Checkpoint 1.4:** Media drive mounted and configured for container user content

---

### Step 1.5: Configure AMP Integration and Game Server Storage

Your system already has AMP (CubeCoders Application Management Panel) running for game server management. This step ensures AMP and Portainer coexist peacefully and configures game servers to use the 4TB media drive for world data.

#### AMP + Portainer Compatibility Overview

**Architecture:**
```
┌─────────────────────────────────────────┐
│  AMP (Native Process)                   │
│  ├── Manages: Game Server Containers   │
│  ├── Features: Mod mgmt, RCON, updates │
│  └── UI: http://localhost:8081  (NEW)  │
│                                         │
│  Portainer (Container)                  │
│  ├── Manages: Infrastructure Containers│
│  ├── Features: Jellyfin, Nextcloud, etc│
│  ├── Visibility: ALL containers         │
│  └── UI: http://localhost:8080  (NEW)  │
│                                         │
│  Both share: /var/run/docker.sock       │
└─────────────────────────────────────────┘
```

**Port Changes:**
- **Portainer**: Now on port 8080 (standard web port)
- **AMP**: Needs to be reconfigured from 8080 → 8081 (we'll do this below)

**Management Strategy:**
- **AMP continues managing:** Game servers (Minecraft, etc.)
- **Portainer manages:** Infrastructure (Jellyfin, Nextcloud, monitoring)
- **Portainer shows:** All containers (including AMP's) for unified visibility
- **No conflicts:** Both tools can safely share Docker socket

#### Reconfigure AMP to Port 8081

Since Portainer now uses port 8080, we need to move AMP to 8081:

```bash
# Stop AMP instances
sudo systemctl stop 'amp*'

# Edit AMP instance manager configuration
# The main instance typically binds to port 8081 in the config already
# but the instances themselves need port changes

# Option 1: Via AMP CLI (recommended)
sudo -u amp ampinstmgr --port 8081

# Option 2: Manual edit (if needed)
# sudo nano /opt/cubecoders/amp/Instances/ADS01/ADS01.kvp
# Find: Core.Webserver.Port=8080
# Change to: Core.Webserver.Port=8081

# Restart AMP
sudo systemctl start ampinstmgr
```

**Verify AMP Port Change:**
```bash
# Check AMP is running on 8081
curl -I http://localhost:8081

# Or check in browser
# Open: http://localhost:8081
```

**Expected Result:** AMP UI loads on port 8081

#### Add User to Docker Group

Allow running docker commands without sudo (matching AMP's access):

```bash
# Add current user to docker group
sudo usermod -aG docker $USER

# Verify (requires logout/login to take effect)
# For now, check the change was made:
grep docker /etc/group
```

**Expected Output:** `docker:x:998:amp,jpmschweitzer`

**Important:** Log out and log back in for group membership to take effect.

#### Functionality Test 1.5A: Verify Docker Access

After logging out and back in:

```bash
# Test docker access without sudo
docker ps

# Should see AMP's containers
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
```

**Expected Results:**
- Can run `docker ps` without sudo
- See AMP's game server containers listed
- No "permission denied" errors

#### Configure AMP to Use Media Drive for Game Data

Currently, AMP stores everything in `/home/amp/.ampdata/`. We'll configure it to store large world data on the 4TB HDD while keeping configs on SSD for performance.

**Storage Strategy for Game Servers:**
- **AMP configs/databases** → `/home/amp/.ampdata/` (SSD - small, frequently accessed)
- **World data/backups** → `/mnt/media/game-servers/` (HDD - large files)

**Option A: Symbolic Links (Recommended)**

Move existing world data to HDD and symlink:

```bash
# Create game server directories on media drive
sudo mkdir -p /mnt/media/game-servers/amp-instances
sudo chown -R amp:amp /mnt/media/game-servers/

# For each AMP instance, move world data (example for one server)
# IMPORTANT: Stop the server in AMP first!
# Then run these commands:

# Example for instance 'justcreate01' - STOP SERVER FIRST IN AMP!
# sudo -u amp mv /home/amp/.ampdata/instances/justcreate01/Minecraft/world \
#   /mnt/media/game-servers/amp-instances/justcreate01-world
#
# sudo -u amp ln -s /mnt/media/game-servers/amp-instances/justcreate01-world \
#   /home/amp/.ampdata/instances/justcreate01/Minecraft/world
```

**Option B: AMP Datastore Configuration (Alternative)**

AMP supports configuring alternate data paths. This can be set per-instance in AMP's web UI:

1. Access AMP: `http://localhost:8080`
2. For each instance:
   - Navigate to instance settings
   - Look for "Data Directory" or "World Path" settings
   - Point to `/mnt/media/game-servers/amp-instances/<instance-name>/`
3. Restart the instance

**Recommendation:** Use Option A (symlinks) as it's more transparent and doesn't require AMP reconfiguration.

#### Configure AMP Backups to Media Drive

```bash
# Create backup directory on media drive
sudo mkdir -p /mnt/media/backups/amp
sudo chown -R amp:amp /mnt/media/backups/amp

# In AMP web UI, configure backup location:
# Settings → Backup → Backup Path: /mnt/media/backups/amp
```

#### Create Directory Structure for Future Game Servers

```bash
# Organized structure for game servers on HDD
mkdir -p /mnt/media/game-servers/amp-instances
mkdir -p /mnt/media/game-servers/minecraft
mkdir -p /mnt/media/game-servers/other
mkdir -p /mnt/media/backups/amp/automated
mkdir -p /mnt/media/backups/amp/manual

# Set ownership for AMP
sudo chown -R amp:amp /mnt/media/game-servers/
sudo chown -R amp:amp /mnt/media/backups/amp/
```

#### Functionality Test 1.5B: Verify AMP Integration

```bash
# Check AMP can access media drive
sudo -u amp touch /mnt/media/game-servers/test-amp-access.txt
sudo -u amp ls -la /mnt/media/game-servers/

# Verify Portainer can see AMP containers
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" | grep -E "Names|amp|minecraft"

# Check disk usage
df -h /mnt/media
df -h /home/amp/.ampdata
```

**Expected Results:**
- AMP user can write to `/mnt/media/game-servers/`
- AMP containers visible in docker ps
- Media drive shows available space
- AMP data directory on SSD still accessible

#### Best Practices for AMP + Portainer Coexistence

**Container Naming Convention:**
- AMP containers: Keep AMP's naming (usually auto-generated)
- Infrastructure containers: Use descriptive names (jellyfin, nextcloud, etc.)

**Management Boundaries:**
- ✅ **Manage via AMP:** Game server containers (start, stop, update, configure)
- ✅ **Manage via Portainer:** Infrastructure containers
- ✅ **View in Portainer:** All containers (for monitoring and unified visibility)
- ❌ **Don't modify AMP containers via Portainer:** Use AMP's UI instead

**In Portainer UI:**
- AMP containers will appear in the container list
- You can view logs and stats
- **Don't stop/restart AMP containers** via Portainer - use AMP UI
- Can tag AMP containers for organization: `game-server`, `amp-managed`

#### Update Phase 1 Completion Checklist

Add AMP-related items to the checklist below.

✅ **Checkpoint 1.5:** AMP integration configured and game server storage optimized

---

### Step 1.6: Deploy Nginx Proxy Manager (Reverse Proxy)

Deploy Nginx Proxy Manager to create a unified web interface consolidating all web-facing services. This provides:
- Single entry point for all services
- Automatic SSL/TLS certificates (Let's Encrypt)
- Easy subdomain/path-based routing
- Web-based configuration interface

**Why Nginx Proxy Manager over Headscale for this?**
- Headscale is an SDN (VPN mesh network) solution - it doesn't handle reverse proxying
- NPM provides a visual interface for managing web traffic routing
- NPM handles SSL certificates automatically
- Both can coexist: Headscale for secure remote access, NPM for unified local/web interface

#### Create Stack in Portainer

1. Navigate to: **Stacks** → **Add Stack**
2. Name: `nginx-proxy-manager`
3. Web editor: Paste the following configuration

#### Docker Compose Configuration

```yaml
version: '3.8'

services:
  nginx-proxy-manager:
    image: jc21/nginx-proxy-manager:latest
    container_name: nginx-proxy-manager
    restart: unless-stopped
    ports:
      - "8000:81"    # Admin web interface (using port 8000)
      - "80:80"      # HTTP traffic
      - "443:443"    # HTTPS traffic
    volumes:
      - /home/jpmschweitzer/docker-data/nginx-proxy-manager/data:/data
      - /home/jpmschweitzer/docker-data/nginx-proxy-manager/letsencrypt:/etc/letsencrypt
    environment:
      - DB_SQLITE_FILE=/data/database.sqlite
```

4. Click **Deploy the stack**

#### Create Directory Structure

```bash
# Create directories for NPM
mkdir -p ~/docker-data/nginx-proxy-manager/data
mkdir -p ~/docker-data/nginx-proxy-manager/letsencrypt

# Set permissions
sudo chown -R $USER:$USER ~/docker-data/nginx-proxy-manager
```

#### Functionality Test 1.6A: Access NPM Web Interface

```bash
# Check NPM container
docker ps | grep nginx-proxy-manager

# Check NPM logs
docker logs nginx-proxy-manager
```

1. Open browser: `http://localhost:8000`
2. First-time login:
   - Email: `admin@example.com`
   - Password: `changeme`
3. **IMPORTANT:** Change admin email and password immediately
4. Dashboard loads showing no proxy hosts yet

**Expected Results:**
- NPM container running
- Can access admin interface on port 8000
- Admin credentials changed

#### Functionality Test 1.6B: Configure Proxy Hosts

Example: Set up Portainer access through NPM

1. In NPM, click **Proxy Hosts** → **Add Proxy Host**
2. Configure:
   - **Domain Names:** `portainer.tower-of-joy.local` (or use your local domain)
   - **Scheme:** `http`
   - **Forward Hostname/IP:** `portainer`
   - **Forward Port:** `9000` (Portainer's internal port)
   - **SSL:** Can enable later with Let's Encrypt
3. Click **Save**

Test access:
- Add to `/etc/hosts` (if using .local): `127.0.0.1 portainer.tower-of-joy.local`
- Access: `http://portainer.tower-of-joy.local`
- Should proxy to Portainer

**Alternative Simple Setup (Path-based):**

If you don't want domain-based routing, you can use path-based routing:
- Keep services on their original ports (8080, 8081, 8096, etc.)
- Access via NPM dashboard on port 8000 as a unified launcher
- Add links in NPM custom HTML page or use Heimdall (Phase 3)

#### Configure Services Through NPM (Optional)

After deploying services in later phases, you can configure:
- `jellyfin.local` → Jellyfin (port 8096)
- `nextcloud.local` → Nextcloud (port 8082)
- `amp.local` → AMP (port 8081)
- `monitor.local` → Uptime Kuma (port 3001)

Or use path-based routing:
- `localhost/jellyfin` → Jellyfin
- `localhost/nextcloud` → Nextcloud
- `localhost/amp` → AMP

#### Functionality Test 1.6C: Verify NPM Working

```bash
# Check NPM is responding
curl -I http://localhost:8000

# Expected: HTTP 200 OK with NPM login page
```

**Expected Results:**
- NPM admin interface accessible on port 8000
- Can create proxy hosts
- Proxy routing works (test with one service)
- Port 8000 now serves as unified web interface entry point

✅ **Checkpoint 1.6:** Nginx Proxy Manager deployed and ready for service routing

---

### Step 1.7: Prepare ML Model Infrastructure (Ollama/vLLM)

Prepare infrastructure for running large language models (LLMs) using Ollama or vLLM with GPU acceleration.

**Hardware Context:**
- **GPU:** NVIDIA RTX 2080 Ti (11GB VRAM, CUDA 11.4)
- **Suitable for:** Models up to ~7B parameters (quantized), some 13B models
- **NVIDIA Container Toolkit:** Already installed in Step 1.1

**Ollama vs vLLM:**
- **Ollama:** Easier to use, built-in model management, good for general use
- **vLLM:** More performant for serving, requires more manual setup, better for production APIs
- **Recommendation:** Start with Ollama (simpler), can add vLLM later if needed

#### Option A: Deploy Ollama (Recommended)

Create directory for Ollama models:

```bash
# Create Ollama directories
mkdir -p ~/docker-data/ollama/models

# Note: Models can be large (4-15GB each)
# Consider moving to HDD if SSD space is limited
# Alternative: mkdir -p /mnt/media/ollama/models
```

Create Stack in Portainer:

1. Navigate to: **Stacks** → **Add Stack**
2. Name: `ollama`
3. Web editor:

```yaml
version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"  # Ollama API
    volumes:
      - /home/jpmschweitzer/docker-data/ollama/models:/root/.ollama  # Model storage
      # Alternative for HDD storage (if SSD space limited):
      # - /mnt/media/ollama/models:/root/.ollama
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```

4. Click **Deploy the stack**

#### Functionality Test 1.7A: Verify Ollama GPU Access

```bash
# Check Ollama container
docker ps | grep ollama

# Check Ollama logs
docker logs ollama

# Verify GPU is accessible inside container
docker exec ollama nvidia-smi
```

**Expected Results:**
- Container running
- GPU visible inside container
- Ollama API listening on port 11434

#### Functionality Test 1.7B: Pull and Test a Model

```bash
# Pull a small model for testing (e.g., Llama 3.2 3B - ~2GB)
docker exec ollama ollama pull llama3.2:3b

# List available models
docker exec ollama ollama list

# Test the model with a simple prompt
docker exec ollama ollama run llama3.2:3b "Hello, how are you?"

# Monitor GPU usage during inference
watch -n 1 nvidia-smi
```

**Expected Results:**
- Model downloads successfully
- Model responds to prompt
- GPU shows utilization during inference
- Response generated in a few seconds

#### Test Ollama API

```bash
# Test API endpoint
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'
```

**Expected:** JSON response with generated text

#### Model Storage Considerations

**VRAM Limits (11GB RTX 2080 Ti):**
- **3B models:** ~2-3GB VRAM (quantized) ✅ Comfortable
- **7B models:** ~4-6GB VRAM (quantized) ✅ Works well
- **13B models:** ~8-10GB VRAM (quantized) ⚠️ Possible, but tight
- **20B+ models:** ❌ Exceeds VRAM, will be very slow

**Disk Storage:**
- Models range from 2GB (3B) to 15GB+ (13B+)
- Recommend starting with 3-7B models
- Store on HDD if SSD space limited (slight load time increase, but inference speed unaffected)

**Recommended Models to Start:**
```bash
# Efficient models for RTX 2080 Ti (11GB VRAM)
docker exec ollama ollama pull llama3.2:3b        # 2GB - Fast, general purpose
docker exec ollama ollama pull mistral:7b         # 4GB - High quality, coding
docker exec ollama ollama pull codellama:7b       # 4GB - Code-specialized
docker exec ollama ollama pull phi3:mini          # 2GB - Fast, reasoning
```

#### Option B: vLLM (Advanced, Optional)

If you need higher performance serving (multiple concurrent requests), you can deploy vLLM:

```yaml
version: '3.8'

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    restart: unless-stopped
    ports:
      - "8001:8000"  # OpenAI-compatible API
    volumes:
      - /home/jpmschweitzer/docker-data/vllm/models:/root/.cache/huggingface
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    command: >
      --model meta-llama/Meta-Llama-3-8B-Instruct
      --dtype float16
      --max-model-len 4096
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```

**Note:** vLLM requires manual model download from HuggingFace and more configuration. Recommend starting with Ollama.

#### Configure NPM Proxy for Ollama (Optional)

Add Ollama to NPM for easier access:
1. NPM → Proxy Hosts → Add
2. Domain: `ollama.tower-of-joy.local` (or path `/ollama`)
3. Forward to: `ollama:11434`
4. Save

#### Functionality Test 1.7C: Performance Validation

```bash
# Test inference speed
time docker exec ollama ollama run llama3.2:3b "Write a haiku about computers"

# Check GPU memory usage
docker exec ollama nvidia-smi

# Verify no CUDA errors in logs
docker logs ollama | grep -i error
```

**Expected Results:**
- Response in 1-5 seconds (depending on model size)
- GPU VRAM usage visible in nvidia-smi
- No CUDA out-of-memory errors
- Smooth inference performance

#### Storage Management for Models

```bash
# Check disk usage
du -sh ~/docker-data/ollama/models/

# If moving to HDD for more space:
# 1. Stop Ollama
# docker stop ollama
#
# 2. Move models
# sudo mv ~/docker-data/ollama/models /mnt/media/ollama/
#
# 3. Update docker-compose volume path
# 4. Restart stack in Portainer
```

✅ **Checkpoint 1.7:** ML model infrastructure ready with GPU acceleration

---

## Phase 1 Completion Checklist

**Infrastructure:**
- [ ] NVIDIA Container Toolkit installed
- [ ] GPU accessible in test container
- [ ] Portainer deployed and running
- [ ] Portainer web UI accessible at port 8080
- [ ] Admin account created
- [ ] Local Docker environment connected
- [ ] GPU management enabled in Portainer

**Storage:**
- [ ] 4TB media drive mounted at /mnt/media
- [ ] Media drive added to /etc/fstab for auto-mount
- [ ] Directory structure created for container content (`/mnt/media/jellyfin/`, `/mnt/media/nextcloud/`, etc.)
- [ ] Permissions set correctly on media drive

**AMP Integration:**
- [ ] AMP reconfigured to port 8081
- [ ] User added to docker group (logout/login required)
- [ ] Can run `docker ps` without sudo
- [ ] AMP containers visible in Portainer
- [ ] Game server directories created on media drive (`/mnt/media/game-servers/`)
- [ ] AMP user has write access to media drive
- [ ] AMP backup directory configured (`/mnt/media/backups/amp/`)
- [ ] Understand AMP + Portainer management boundaries

**Reverse Proxy:**
- [ ] Nginx Proxy Manager deployed and running
- [ ] NPM web UI accessible at port 8000
- [ ] Admin credentials changed from defaults
- [ ] Can create proxy hosts
- [ ] Port 8000 serving as unified web interface entry point

**ML Infrastructure:**
- [ ] Ollama deployed with GPU support
- [ ] Ollama API accessible at port 11434
- [ ] GPU visible inside Ollama container
- [ ] Test model pulled and working (llama3.2:3b or similar)
- [ ] GPU acceleration confirmed during inference
- [ ] Model storage configured (SSD or HDD based on space)

**Phase 1 Status:** Foundation established with container management, dual-disk storage, AMP integration, reverse proxy, and ML infrastructure

---

## Phase 2: Networking & External Access

**Goal:** Deploy Headscale for secure external access
**Estimated Time:** 2-6 hours
**Prerequisites:** Phase 1 completed successfully

### Step 2.1: Deploy Headscale

Headscale provides a self-hosted Tailscale control server for secure mesh networking.

#### Create Stack in Portainer

1. Navigate to: **Stacks** → **Add Stack**
2. Name: `headscale`
3. Web editor: Paste the following configuration

#### Docker Compose Configuration

```yaml
version: '3.8'

services:
  headscale:
    image: headscale/headscale:latest
    container_name: headscale
    restart: unless-stopped
    ports:
      - "8085:8080"  # Web/API port
      - "9090:9090"  # Metrics port (optional)
    volumes:
      - /home/jpmschweitzer/docker-data/headscale/config:/etc/headscale
      - /home/jpmschweitzer/docker-data/headscale/data:/var/lib/headscale
    command: headscale serve
    networks:
      - headscale-network

networks:
  headscale-network:
    driver: bridge
```

4. Click **Deploy the stack**

#### Functionality Test 3.1A: Verify Headscale Running

```bash
# Check Headscale container
docker ps | grep headscale

# Check Headscale logs
docker logs headscale
```

**Expected Results:**
- Container running
- Logs show Headscale server starting
- Listening on port 8080 (inside container)

#### Step 2.1B: Initialize Headscale Configuration

```bash
# Create config directory
mkdir -p ~/docker-data/headscale/config
mkdir -p ~/docker-data/headscale/data

# Generate default config
docker exec headscale headscale config generate > ~/docker-data/headscale/config/config.yaml

# Edit config (important settings)
nano ~/docker-data/headscale/config/config.yaml
```

**Key Configuration Changes:**

```yaml
# Change server URL to your server IP or hostname
server_url: http://tower-of-joy:8085

# Or use your local IP
server_url: http://192.168.x.x:8085

# Set database path
db_type: sqlite3
db_path: /var/lib/headscale/db.sqlite

# Enable private key generation
private_key_path: /var/lib/headscale/private.key
```

Save and restart Headscale:

```bash
# Restart Headscale to apply config
docker restart headscale

# Wait a few seconds and check logs
docker logs headscale
```

#### Functionality Test 3.1B: Verify Headscale API

```bash
# Test API access
curl http://localhost:8085/health
```

**Expected Output:** `{"healthy":true}` or similar health check response

✅ **Checkpoint 2.1:** Headscale server running and accessible

---

### Step 2.2: Create Headscale User and Pre-Auth Key

#### Commands

```bash
# Create a user (namespace) in Headscale
docker exec headscale headscale users create homelab

# Generate a pre-authentication key (for easy device enrollment)
docker exec headscale headscale preauthkeys create --user homelab --expiration 24h

# Save this key - you'll need it for connecting devices
```

**Expected Output:** A pre-auth key like `abc123def456...`

#### Functionality Test 3.2: List Users

```bash
# Verify user created
docker exec headscale headscale users list
```

**Expected Output:** Shows `homelab` user

✅ **Checkpoint 2.2:** Headscale user and pre-auth key created

---

### Step 2.3: Connect First Device (This Server)

Install Tailscale client on the host machine and connect to Headscale.

#### Commands

```bash
# Install Tailscale client
curl -fsSL https://tailscale.com/install.sh | sh

# Connect to Headscale server
sudo tailscale up --login-server=http://localhost:8085 --authkey=<your-preauthkey-here>
```

Replace `<your-preauthkey-here>` with the key from Step 3.2

#### Functionality Test 3.3: Verify Connection

```bash
# Check Tailscale status
tailscale status

# Get this machine's Tailscale IP
tailscale ip -4

# On Headscale, list nodes
docker exec headscale headscale nodes list
```

**Expected Results:**
- Tailscale shows connected
- Machine has Tailscale IP (e.g., 100.64.0.1)
- Headscale nodes list shows this machine

✅ **Checkpoint 2.3:** Server connected to Headscale network

---

### Step 2.4: Connect Additional Devices

Connect your laptop, phone, or other devices to access services remotely.

#### For Linux/Mac/Windows Desktop

1. Install Tailscale: https://tailscale.com/download
2. Run: `tailscale up --login-server=http://tower-of-joy:8085`
3. Browser opens to registration page
4. Generate pre-auth key on server:
   ```bash
   docker exec headscale headscale preauthkeys create --user homelab --expiration 24h
   ```
5. Use the registration URL shown, or use the pre-auth key

#### For Mobile (iOS/Android)

1. Install Tailscale app from App Store/Play Store
2. Open app → Settings → Use custom control URL
3. Enter: `http://<your-server-ip>:8085`
4. Use pre-auth key or registration URL

#### Functionality Test 3.4: Test Device Connectivity

From the newly connected device:

```bash
# Ping the server's Tailscale IP
ping <server-tailscale-ip>

# Try accessing Jellyfin via Tailscale
# Open browser to: http://<server-tailscale-ip>:8096
```

**Expected Results:**
- Can ping server via Tailscale IP
- Can access Jellyfin web interface via Tailscale
- Can access Nextcloud via Tailscale

✅ **Checkpoint 2.4:** Remote devices can access services via Headscale/Tailscale

---

### Step 2.5: Configure Tailscale Serve (Optional)

Tailscale Serve provides HTTPS access to services with automatic certificates.

```bash
# Serve Jellyfin on HTTPS
sudo tailscale serve https / http://localhost:8096

# Or serve multiple services on different paths
sudo tailscale serve https /jellyfin http://localhost:8096
sudo tailscale serve https /nextcloud http://localhost:8080
```

#### Functionality Test 3.5: Access Services via Tailscale HTTPS

From any device on Tailscale network:
1. Get server's Tailscale hostname: `tailscale status`
2. Access in browser: `https://<machine-name>.tailnet-name.ts.net/jellyfin`

**Expected Results:**
- HTTPS connection (valid certificate)
- Service loads correctly
- No certificate warnings

✅ **Checkpoint 2.5:** HTTPS access configured via Tailscale Serve

---

## Phase 2 Completion Checklist

- [ ] Headscale server deployed and running
- [ ] Headscale user created
- [ ] Pre-auth keys generated
- [ ] Server connected to Headscale
- [ ] Additional devices connected
- [ ] Can access services remotely via Tailscale IPs
- [ ] (Optional) Tailscale Serve configured for HTTPS

**Phase 2 Status:** Secure remote access operational

---

## Phase 3: Monitoring & Management

**Goal:** Add monitoring, dashboards, and management tools
**Estimated Time:** 2-4 hours
**Prerequisites:** Phase 2 completed successfully

### Step 3.1: Deploy Uptime Kuma (Service Monitoring)

Uptime Kuma monitors service availability and sends notifications.

#### Create Stack in Portainer

1. Navigate to: **Stacks** → **Add Stack**
2. Name: `uptime-kuma`
3. Web editor:

```yaml
version: '3.8'

services:
  uptime-kuma:
    image: louislam/uptime-kuma:latest
    container_name: uptime-kuma
    restart: unless-stopped
    ports:
      - "3001:3001"
    volumes:
      - /home/jpmschweitzer/docker-data/uptime-kuma:/app/data
```

4. Deploy the stack

#### Functionality Test 4.1: Configure Uptime Kuma

1. Open browser: `http://localhost:3001`
2. Create admin account
3. Add monitors for:
   - Jellyfin: `http://localhost:8096`
   - Nextcloud: `http://localhost:8080`
   - Portainer: `http://localhost:9000`
   - Headscale: `http://localhost:8085/health`
4. Set check intervals (e.g., 60 seconds)

**Expected Results:**
- All monitors showing "Up" status
- Dashboard displays service uptime

✅ **Checkpoint 3.1:** Service monitoring operational

---

### Step 3.2: Deploy Netdata (System Monitoring)

Netdata provides real-time system performance monitoring.

#### Create Stack in Portainer

```yaml
version: '3.8'

services:
  netdata:
    image: netdata/netdata:latest
    container_name: netdata
    restart: unless-stopped
    hostname: tower-of-joy
    ports:
      - "19999:19999"
    cap_add:
      - SYS_PTRACE
    security_opt:
      - apparmor:unconfined
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - NETDATA_CLAIM_TOKEN=  # Optional: claim token for Netdata Cloud
      - NETDATA_CLAIM_URL=https://app.netdata.cloud
```

Deploy the stack.

#### Functionality Test 4.2: Access Netdata Dashboard

1. Open browser: `http://localhost:19999`
2. Explore dashboard sections:
   - CPU usage
   - RAM usage
   - Disk I/O
   - Network traffic
   - Docker containers
   - GPU monitoring (if configured)

**Expected Results:**
- Real-time graphs updating
- All system metrics visible
- Docker containers monitored

✅ **Checkpoint 3.2:** System monitoring operational

---

### Step 3.3: Deploy Heimdall (Application Dashboard)

Heimdall provides a unified dashboard with links to all services.

#### Create Stack in Portainer

```yaml
version: '3.8'

services:
  heimdall:
    image: linuxserver/heimdall:latest
    container_name: heimdall
    restart: unless-stopped
    ports:
      - "8888:80"
      - "8889:443"
    volumes:
      - /home/jpmschweitzer/docker-data/heimdall:/config
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Amsterdam
```

Deploy the stack.

#### Functionality Test 4.3: Configure Heimdall Dashboard

1. Open browser: `http://localhost:8888`
2. Add applications:
   - Jellyfin: `http://tower-of-joy:8096`
   - Nextcloud: `http://tower-of-joy:8080`
   - Portainer: `http://tower-of-joy:9000`
   - Uptime Kuma: `http://tower-of-joy:3001`
   - Netdata: `http://tower-of-joy:19999`
3. Customize colors and icons

**Expected Results:**
- Unified dashboard showing all services
- One-click access to any service
- Clean, organized interface

✅ **Checkpoint 3.3:** Application dashboard operational

---

## Phase 3 Completion Checklist

- [ ] Uptime Kuma monitoring all services
- [ ] Netdata showing real-time system metrics
- [ ] Heimdall dashboard with all service links
- [ ] All dashboards accessible via Tailscale

**Phase 3 Status:** Monitoring and management tools operational

---

## Phase 4: Optimization & Security

**Goal:** Harden security and optimize performance
**Estimated Time:** 2-6 hours
**Prerequisites:** Phase 3 completed successfully

### Step 4.1: Configure Automatic Container Updates

Use Watchtower to automatically update containers.

#### Create Stack in Portainer

```yaml
version: '3.8'

services:
  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_SCHEDULE=0 0 4 * * *  # 4 AM daily
      - WATCHTOWER_NOTIFICATIONS=shoutrrr
      - WATCHTOWER_NOTIFICATION_URL=  # Optional: notification URL
```

Deploy the stack.

#### Functionality Test 5.1: Verify Watchtower

```bash
# Check Watchtower logs
docker logs watchtower

# Force a scan
docker exec watchtower watchtower --run-once
```

**Expected Results:**
- Watchtower checks all containers
- Updates available containers (if any)
- Logs show scan results

✅ **Checkpoint 4.1:** Automatic updates configured

---

### Step 4.2: Configure Docker Log Rotation

Prevent logs from consuming disk space.

#### Commands

```bash
# Create Docker daemon config
sudo nano /etc/docker/daemon.json
```

Add the following:

```json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
```

Save and restart Docker:

```bash
sudo systemctl restart docker
```

#### Functionality Test 5.2: Verify Log Configuration

```bash
# Check Docker info
docker info | grep -A 5 "Logging Driver"

# Restart a container to apply new log settings
docker restart jellyfin

# Check log file sizes
sudo du -sh /var/lib/docker/containers/*/*-json.log
```

**Expected Results:**
- Log driver shows json-file with size limits
- Log files stay under 10MB

✅ **Checkpoint 4.2:** Log rotation configured

---

### Step 4.3: Configure Firewall Rules

Set up UFW (Uncomplicated Firewall) to secure the server.

#### Commands

```bash
# Install UFW if not installed
sudo apt-get install -y ufw

# Allow SSH (IMPORTANT - do this first!)
sudo ufw allow 22/tcp

# Allow Tailscale
sudo ufw allow 41641/udp

# Allow Portainer
sudo ufw allow 9000/tcp

# Allow Jellyfin (only if you want LAN access without Tailscale)
sudo ufw allow 8096/tcp

# Allow Nextcloud (only if you want LAN access without Tailscale)
sudo ufw allow 8080/tcp

# Allow Samba (only if you want network shares)
sudo ufw allow 139/tcp
sudo ufw allow 445/tcp

# Enable firewall
sudo ufw enable

# Check status
sudo ufw status verbose
```

#### Functionality Test 5.3: Verify Firewall

```bash
# Check UFW status
sudo ufw status numbered

# Test accessing services from another device
# Should work: Portainer, Jellyfin, Nextcloud (if allowed)
# Should work via Tailscale regardless of firewall
```

**Expected Results:**
- UFW enabled and active
- Allowed services accessible
- Tailscale connections work

✅ **Checkpoint 4.3:** Firewall configured and active

---

### Step 4.4: Set Up Backup Strategy

Configure backups for critical data.

#### Deploy Duplicati (Backup Solution)

```yaml
version: '3.8'

services:
  duplicati:
    image: linuxserver/duplicati:latest
    container_name: duplicati
    restart: unless-stopped
    ports:
      - "8200:8200"
    volumes:
      - /home/jpmschweitzer/docker-data/duplicati/config:/config
      - /home/jpmschweitzer/docker-data:/source/docker-data:ro
      - /path/to/backup/destination:/backups
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Amsterdam
```

Deploy and configure:

1. Access Duplicati: `http://localhost:8200`
2. Create backup job:
   - Source: `/source/docker-data`
   - Destination: Local folder, external drive, or cloud (S3, Google Drive, etc.)
   - Schedule: Daily at 3 AM
   - Retention: Keep 30 days
3. Run test backup

#### Functionality Test 5.4: Verify Backup

```bash
# Check Duplicati logs
docker logs duplicati

# Verify backup files created
ls -lh /path/to/backup/destination
```

**Expected Results:**
- Backup job completes successfully
- Backup files created
- Can restore test file

✅ **Checkpoint 4.4:** Backup system operational

---

### Step 4.5: Optimize Nextcloud Performance

Apply recommended Nextcloud optimizations.

#### Commands

```bash
# Install missing indices
docker exec -u www-data nextcloud php occ db:add-missing-indices

# Convert to bigint (if needed)
docker exec -u www-data nextcloud php occ db:convert-filecache-bigint

# Configure memory cache
docker exec -u www-data nextcloud php occ config:system:set memcache.local --value="\\OC\\Memcache\\Redis"
docker exec -u www-data nextcloud php occ config:system:set memcache.locking --value="\\OC\\Memcache\\Redis"
docker exec -u www-data nextcloud php occ config:system:set redis host --value="nextcloud-redis"
docker exec -u www-data nextcloud php occ config:system:set redis port --value=6379

# Set up cron instead of AJAX
docker exec -u www-data nextcloud php occ background:cron

# Add to crontab (on host)
echo "*/5 * * * * docker exec -u www-data nextcloud php cron.php" | sudo tee -a /etc/crontab
```

#### Functionality Test 5.5: Verify Optimizations

1. Access Nextcloud: **Settings** → **Overview**
2. Check for warnings/errors
3. Upload/download test files to check performance

**Expected Results:**
- No warnings in Overview
- Background job set to Cron
- Improved performance

✅ **Checkpoint 4.5:** Nextcloud optimized

---

## Phase 4 Completion Checklist

- [ ] Watchtower configured for automatic updates
- [ ] Docker log rotation enabled
- [ ] Firewall rules configured
- [ ] Backup solution deployed and tested
- [ ] Nextcloud optimizations applied
- [ ] Security best practices implemented

**Phase 4 Status:** System secured and optimized

---

## Post-Implementation Verification

### Complete System Test

Run these tests to verify the entire system:

#### 1. Service Availability Test

```bash
# Check all containers running
docker ps

# Expected: All containers showing "Up" status
```

#### 2. GPU Transcoding Test

1. Access Jellyfin
2. Start playing video that requires transcoding
3. Monitor GPU: `watch nvidia-smi`
4. Verify Video Engine usage

#### 3. Remote Access Test

1. From device on Tailscale network
2. Access Jellyfin via Tailscale IP
3. Stream video remotely
4. Verify smooth playback

#### 4. File Sharing Test

1. From another computer
2. Connect to `\\tower-of-joy\Media`
3. Copy file to share
4. Verify successful transfer

#### 5. Nextcloud Sync Test

1. Install Nextcloud desktop client
2. Connect via Tailscale IP
3. Sync test folder
4. Verify files sync correctly

#### 6. Monitoring Test

1. Access Uptime Kuma
2. Verify all services showing "Up"
3. Access Netdata
4. Verify system metrics displaying

#### 7. Backup Test

1. Access Duplicati
2. Verify last backup successful
3. Test file restore
4. Confirm restore works

### Final Checklist

- [ ] All containers running
- [ ] GPU transcoding working
- [ ] Remote access via Tailscale functional
- [ ] File sharing accessible
- [ ] Nextcloud syncing
- [ ] Monitoring operational
- [ ] Backups running
- [ ] Firewall active
- [ ] Log rotation enabled
- [ ] Automatic updates configured

---

## Troubleshooting Guide

### Common Issues and Solutions

#### Issue: GPU Not Detected in Container

**Symptoms:**
- nvidia-smi fails in container
- Jellyfin shows no hardware encoding options

**Solutions:**
```bash
# Verify NVIDIA runtime
docker info | grep -i runtime

# Restart Docker
sudo systemctl restart docker

# Verify GPU access
docker run --rm --gpus all nvidia/cuda:11.4.0-base-ubuntu20.04 nvidia-smi
```

---

#### Issue: Container Won't Start

**Symptoms:**
- Container status: Exited
- Service not accessible

**Solutions:**
```bash
# Check logs
docker logs <container-name>

# Check for port conflicts
sudo netstat -tulpn | grep <port>

# Verify volume permissions
ls -la ~/docker-data/<service>
```

---

#### Issue: Out of Disk Space

**Symptoms:**
- Containers failing to start
- df shows high usage

**Solutions:**
```bash
# Clean up unused containers/images
docker system prune -a

# Check Docker disk usage
docker system df

# Move media to external drive
# Update volume mounts in docker-compose
```

---

#### Issue: Can't Access via Tailscale

**Symptoms:**
- Can't ping Tailscale IP
- Services not accessible remotely

**Solutions:**
```bash
# Check Tailscale status
tailscale status

# Restart Tailscale
sudo systemctl restart tailscaled

# Check Headscale nodes
docker exec headscale headscale nodes list

# Verify firewall not blocking
sudo ufw status
```

---

#### Issue: Jellyfin Transcoding Using CPU Instead of GPU

**Symptoms:**
- High CPU usage during playback
- Dashboard shows software transcoding

**Solutions:**
1. Jellyfin Dashboard → Playback → Transcoding
2. Verify NVIDIA NVENC selected
3. Check all hardware decoding options enabled
4. Restart Jellyfin container
5. Test with compatible video format (H.264)

---

#### Issue: Nextcloud Slow Performance

**Symptoms:**
- Slow file uploads
- Sluggish web interface

**Solutions:**
```bash
# Verify Redis connected
docker logs nextcloud | grep -i redis

# Check cron job active
docker exec -u www-data nextcloud php occ config:system:get maintenance_window_start

# Add missing indices
docker exec -u www-data nextcloud php occ db:add-missing-indices

# Increase memory limits
docker exec -u www-data nextcloud php occ config:system:set memory_limit --value="512M"
```

---

## Maintenance Tasks

### Daily
- [ ] Check Uptime Kuma dashboard for service status
- [ ] Monitor disk space: `df -h`

### Weekly
- [ ] Review container logs for errors
- [ ] Check backup completion in Duplicati
- [ ] Review Netdata for unusual activity

### Monthly
- [ ] Update containers (if not using Watchtower)
- [ ] Review and clean old Docker images: `docker image prune -a`
- [ ] Test backup restoration
- [ ] Review firewall logs: `sudo ufw status verbose`

### Quarterly
- [ ] Update system packages: `sudo apt update && sudo apt upgrade`
- [ ] Review container performance and resource usage
- [ ] Audit Tailscale/Headscale connected devices
- [ ] Review and update documentation

---

## Performance Optimization Tips

### For Limited Storage (481GB):

1. **Use external drives for media**
   - Mount external HDD/SSD
   - Update Jellyfin volume mounts
   - Keeps system drive lean

2. **Enable aggressive log rotation**
   - Current config: 10MB × 3 files per container
   - Can reduce to 5MB × 2 files if needed

3. **Regular cleanup**
   - Schedule weekly: `docker system prune -a --volumes --filter "until=168h"`
   - Removes unused images/volumes older than 7 days

4. **Monitor disk usage**
   - Add Netdata disk space alerts
   - Set threshold at 85% usage

### For Better Performance:

1. **Optimize container resource limits**
   - Add resource limits in docker-compose
   - Prevent any container from consuming all resources

2. **Use SSD for Docker data**
   - If possible, move `/var/lib/docker` to SSD
   - Significantly improves container performance

3. **Enable Docker experimental features**
   - Better caching
   - Faster builds

4. **Tune Jellyfin transcoding**
   - Limit concurrent transcodes
   - Prefer direct play when possible
   - Use hardware-compatible codecs (H.264/H.265)

---

## Expansion Ideas

### Future Services to Consider:

**AI/ML Integration:**
1. **Open WebUI** - Web interface for Ollama (like ChatGPT UI)
2. **Stable Diffusion WebUI** - GPU-accelerated image generation
3. **ComfyUI** - Node-based interface for AI workflows
4. **Whisper** - Speech-to-text transcription service
5. **Text Generation WebUI** - Alternative LLM interface

**Productivity & Media:**
6. **Pi-hole** - Network-wide ad blocking
7. **Vaultwarden** - Self-hosted password manager
8. **Home Assistant** - Home automation platform
9. **Gitea** - Self-hosted Git service
10. **Paperless-ngx** - Document management
11. **Calibre-Web** - Ebook library
12. **Audiobookshelf** - Audiobook server
13. **Photoprism** - Photo management
14. **Grafana** - Advanced metrics visualization

### Hardware Upgrades:

1. **UPS** - Uninterruptible power supply for clean shutdowns and graceful shutdown on power loss
2. **Network upgrade** - 2.5GbE or 10GbE NIC for faster media streaming and transfers
3. **More RAM** - 32GB for running more simultaneous containers
4. **Additional storage** - Optional: More HDDs if 3.7TB isn't enough (current setup has plenty)
5. **Backup drive** - External USB drive for offsite backups

---

## Documentation & Resources

### Container Configuration Files

All Docker Compose files are managed in Portainer Stacks:
- Portainer UI: `http://tower-of-joy:9000`
- Stacks location: Portainer → Stacks
- Export stacks regularly for backup

### Important Paths

**Storage Strategy:**
- **SSD (/dev/sda)**: Configs, databases, cache (performance-critical)
- **HDD (/dev/sdb)**: User content, media, bulk storage (capacity-critical)

| Service | Config Path (SSD) | Data Path (HDD) |
|---------|-------------------|-----------------|
| **Portainer** | Docker volume: `portainer_data` | N/A |
| **Nginx Proxy Manager** | `~/docker-data/nginx-proxy-manager/data` | N/A |
| **NPM SSL Certs** | `~/docker-data/nginx-proxy-manager/letsencrypt` | N/A |
| **Ollama Models** | `~/docker-data/ollama/models` | Alternative: `/mnt/media/ollama/models` |
| **AMP (Game Servers)** | `/home/amp/.ampdata/` | `/mnt/media/game-servers/amp-instances/` |
| AMP Backups | N/A | `/mnt/media/backups/amp/` |
| Headscale | `~/docker-data/headscale/config` | `~/docker-data/headscale/data` |
| Jellyfin | `~/docker-data/jellyfin/config` | `/mnt/media/jellyfin/` |
| Nextcloud | `~/docker-data/nextcloud/config` | `/mnt/media/nextcloud/data/` |
| Nextcloud DB | `~/docker-data/nextcloud/db` | N/A (DB on SSD) |
| Samba | `~/docker-data/samba` | `/mnt/media/` (shares) |
| Backups | N/A | `/mnt/media/backups/` |
| Future Game Servers | `~/docker-data/<game>/config` | `/mnt/media/game-servers/<game>/` |
| vLLM Models (optional) | `~/docker-data/vllm/models` | Alternative: `/mnt/media/vllm/models` |

### Network Ports

| Service | Port | Access | Notes |
|---------|------|--------|-------|
| **Nginx Proxy Manager** | 8000 | HTTP | **Unified web interface** - Phase 1 |
| **Nginx Proxy Manager** | 80 | HTTP | **Reverse proxy traffic** - Phase 1 |
| **Nginx Proxy Manager** | 443 | HTTPS | **Reverse proxy SSL** - Phase 1 |
| **AMP (Game Servers)** | 8081 | HTTP | Reconfigured from 8080 - Phase 1 |
| **Portainer** | 8080 | HTTP | Container management - Phase 1 |
| **Portainer** | 8443 | HTTPS | Portainer HTTPS - Phase 1 |
| **Ollama** | 11434 | HTTP | ML model API - Phase 1 |
| Headscale | 8085 | HTTP | SDN control server - Phase 2 |
| Headscale | 41641 | UDP | Tailscale mesh - Phase 2 |
| Uptime Kuma | 3001 | HTTP | Service monitoring - Phase 3 |
| Netdata | 19999 | HTTP | System monitoring - Phase 3 |
| Heimdall | 8888 | HTTP | Application dashboard - Phase 3 |
| Duplicati | 8200 | HTTP | Backup management - Phase 4 |
| Jellyfin | 8096 | HTTP | Media server - Backlog |
| Nextcloud | 8082 | HTTP | Cloud storage - Backlog |
| Samba | 445 | SMB | File sharing - Backlog |
| Minecraft Servers | 25565+ | TCP/UDP | Managed by AMP |
| vLLM (optional) | 8001 | HTTP | ML model API (OpenAI-compatible) |

### Support Resources

**Infrastructure:**
- **Portainer Docs**: https://docs.portainer.io/
- **Nginx Proxy Manager**: https://nginxproxymanager.com/guide/
- **Docker Docs**: https://docs.docker.com/
- **NVIDIA Container Toolkit**: https://docs.nvidia.com/datacenter/cloud-native/

**ML/AI:**
- **Ollama Docs**: https://github.com/ollama/ollama
- **vLLM Docs**: https://docs.vllm.ai/
- **Ollama Library**: https://ollama.com/library (available models)

**Networking:**
- **Headscale Docs**: https://headscale.net/
- **Tailscale KB**: https://tailscale.com/kb/

**Applications:**
- **Jellyfin Docs**: https://jellyfin.org/docs/
- **Nextcloud Admin Manual**: https://docs.nextcloud.com/

---

## Implementation Complete!

🎉 **Congratulations!** Your home server is now fully operational with:

**Phase 1 - Foundation:**
- ✅ Web-based container management (Portainer)
- ✅ Unified reverse proxy interface (Nginx Proxy Manager)
- ✅ GPU-accelerated ML models (Ollama/vLLM)
- ✅ Dual-disk storage strategy (SSD + 4TB HDD)
- ✅ AMP game server integration

**Phase 2 - Networking:**
- ✅ Secure remote access (Headscale/Tailscale)
- ✅ SDN mesh networking

**Phase 3 - Monitoring:**
- ✅ Service monitoring (Uptime Kuma)
- ✅ System monitoring (Netdata)
- ✅ Unified dashboard (Heimdall)

**Phase 4 - Optimization:**
- ✅ Automatic updates (Watchtower)
- ✅ Backup solution (Duplicati)
- ✅ Security hardening (UFW)
- ✅ Log rotation

**Backlog - Applications:**
- ✅ Media server with GPU transcoding (Jellyfin)
- ✅ Cloud storage with external access (Nextcloud)
- ✅ Network file sharing (Samba)

**Total Services Running:** 13+ containers
**Total Setup Time:** 8-20 hours (including ML setup)
**System Status:** Production-ready with ML capabilities

### Next Steps:

1. Customize services to your preferences
2. Add media to Jellyfin libraries
3. Upload files to Nextcloud
4. Connect all your devices to Tailscale
5. Set up mobile apps
6. Explore additional services from expansion ideas
7. Enjoy your self-hosted home server!

---

*Implementation Plan Version 1.0 - Last Updated: 2025-11-11*

---

## Phase Backlog: Application Deployment

**Status:** Deferred until base infrastructure (Phases 1-4) is complete

The following application deployments will be implemented after the base infrastructure is solid. This includes deploying Jellyfin media server with GPU transcoding, Nextcloud cloud storage, and Samba file sharing.

---

## Phase 2: Core Services Deployment

**Goal:** Deploy Jellyfin, Nextcloud, and file sharing services
**Estimated Time:** 2-4 hours
**Prerequisites:** Base infrastructure completed (Phases 1-4)

**Note:** This phase is in the backlog and will be implemented after the base infrastructure is solid.

---

### Step 2.1: Prepare Directory Structure

Create directories for container configurations and user content.

**Storage Strategy Reminder:**
- **Container configs** → `~/docker-data/*` (SSD - fast access)
- **User content** → `/mnt/media/*` (4TB HDD - bulk storage)

#### Commands

```bash
# Create base directory for container configs on SSD
mkdir -p ~/docker-data

# Create Jellyfin config directories (SSD for configs, HDD for media)
mkdir -p ~/docker-data/jellyfin/config
mkdir -p ~/docker-data/jellyfin/cache

# Create Nextcloud config directory (SSD)
mkdir -p ~/docker-data/nextcloud/config
mkdir -p ~/docker-data/nextcloud/db

# Create Samba config directory (SSD)
mkdir -p ~/docker-data/samba/config

# Set permissions on SSD directories
sudo chown -R $USER:$USER ~/docker-data

# Verify media drive directories from Phase 1
# These should already exist from Step 1.4
ls -la /mnt/media

# If needed, create additional media subdirectories
mkdir -p /mnt/media/jellyfin/movies
mkdir -p /mnt/media/jellyfin/tv
mkdir -p /mnt/media/jellyfin/music
mkdir -p /mnt/media/nextcloud/data
```

#### Functionality Test 2.1: Verify Directory Structure

```bash
# Check SSD directories
tree -L 3 ~/docker-data

# Check HDD directories
tree -L 2 /mnt/media

# Verify permissions
ls -la ~/docker-data
ls -la /mnt/media

# Verify media drive is mounted
df -h /mnt/media
```

**Expected Results:**
- SSD config directories created in `~/docker-data/`
- HDD content directories exist in `/mnt/media/`
- All directories owned by current user
- Media drive showing 3.7TB available
- Write permissions enabled on both locations

✅ **Checkpoint 2.1:** Directory structure prepared with proper SSD/HDD separation

---

### Step 2.2: Deploy Jellyfin Media Server with GPU

Deploy Jellyfin with NVIDIA GPU support for hardware transcoding.

#### Create Stack in Portainer

1. Navigate to: **Stacks** → **Add Stack**
2. Name: `jellyfin`
3. Web editor: Paste the following Docker Compose configuration

#### Docker Compose Configuration

```yaml
version: '3.8'

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    user: 1000:1000  # Replace with your UID:GID (run `id` to find)
    network_mode: bridge
    ports:
      - 8096:8096      # HTTP web interface
      - 8920:8920      # HTTPS web interface (optional)
      - 7359:7359/udp  # Auto-discovery
      - 1900:1900/udp  # DLNA
    volumes:
      - /home/jpmschweitzer/docker-data/jellyfin/config:/config  # Config on SSD
      - /home/jpmschweitzer/docker-data/jellyfin/cache:/cache    # Cache on SSD
      - /mnt/media/jellyfin:/media:ro  # Media on 4TB HDD (read-only)
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu, video, compute, utility]
```

4. Click **Deploy the stack**

#### Functionality Test 2.2A: Verify Jellyfin Container Running

```bash
# Check Jellyfin container status
docker ps | grep jellyfin

# Check Jellyfin logs
docker logs jellyfin

# Verify GPU access inside container
docker exec jellyfin nvidia-smi
```

**Expected Results:**
- Container status: "Up"
- Logs show Jellyfin starting without errors
- `nvidia-smi` inside container shows RTX 2080 Ti

#### Functionality Test 2.2B: Access Jellyfin Web Interface

1. Open browser: `http://localhost:8096` or `http://tower-of-joy:8096`
2. Complete initial setup wizard:
   - Choose language
   - Create admin account
   - Add media libraries (point to /media/movies, /media/tv, etc.)
3. Navigate to: **Dashboard** → **Playback** → **Transcoding**
4. Enable hardware acceleration:
   - Hardware acceleration: **NVIDIA NVENC**
   - Enable hardware decoding for: (select all applicable)
   - Enable hardware encoding
   - Enable NVENC
   - Encoding preset: Auto or High Quality

#### Functionality Test 2.2C: Test GPU Transcoding

1. Upload a test video (or add existing media)
2. Start playback in browser
3. In Dashboard → Activity, watch transcoding session
4. On terminal, monitor GPU usage:

```bash
# Watch GPU utilization during transcoding
watch -n 1 nvidia-smi
```

**Expected Results:**
- Video plays smoothly
- Dashboard shows "(hw)" next to codec name during transcode
- `nvidia-smi` shows GPU utilization (Video Engine usage)
- CPU usage remains low during transcoding

✅ **Checkpoint 2.2:** Jellyfin deployed with working GPU hardware transcoding

---

### Step 2.3: Deploy Nextcloud with Database

Deploy Nextcloud with MariaDB and Redis for optimal performance.

#### Create Stack in Portainer

1. Navigate to: **Stacks** → **Add Stack**
2. Name: `nextcloud`
3. Web editor: Paste the following configuration

#### Docker Compose Configuration

```yaml
version: '3.8'

services:
  nextcloud-db:
    image: mariadb:10.11
    container_name: nextcloud-db
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    restart: unless-stopped
    volumes:
      - /home/jpmschweitzer/docker-data/nextcloud/db:/var/lib/mysql  # Database on SSD (performance-critical)
    environment:
      - MYSQL_ROOT_PASSWORD=changeme_root_password  # CHANGE THIS!
      - MYSQL_PASSWORD=changeme_nc_password         # CHANGE THIS!
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    networks:
      - nextcloud-network

  nextcloud-redis:
    image: redis:alpine
    container_name: nextcloud-redis
    restart: unless-stopped
    networks:
      - nextcloud-network

  nextcloud:
    image: nextcloud:stable
    container_name: nextcloud
    restart: unless-stopped
    ports:
      - 8080:80
    volumes:
      - /home/jpmschweitzer/docker-data/nextcloud/config:/var/www/html  # App config on SSD
      - /mnt/media/nextcloud/data:/var/www/html/data  # User data on 4TB HDD
    environment:
      - MYSQL_HOST=nextcloud-db
      - MYSQL_PASSWORD=changeme_nc_password  # Match DB password
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - REDIS_HOST=nextcloud-redis
    depends_on:
      - nextcloud-db
      - nextcloud-redis
    networks:
      - nextcloud-network

networks:
  nextcloud-network:
    driver: bridge
```

**IMPORTANT:** Change the database passwords before deploying!

4. Click **Deploy the stack**

#### Functionality Test 2.3A: Verify Nextcloud Stack Running

```bash
# Check all Nextcloud containers
docker ps | grep nextcloud

# Check Nextcloud logs
docker logs nextcloud

# Check database logs
docker logs nextcloud-db
```

**Expected Results:**
- All three containers running (nextcloud, nextcloud-db, nextcloud-redis)
- No error messages in logs
- Database initialized successfully

#### Functionality Test 2.3B: Complete Nextcloud Setup

1. Open browser: `http://localhost:8080` or `http://tower-of-joy:8080`
2. First-time setup page appears
3. Create admin account:
   - Username: (choose admin username)
   - Password: (strong password)
4. Storage & database section:
   - Data folder: Leave default `/var/www/html/data`
   - Database: **MySQL/MariaDB**
   - Database user: `nextcloud`
   - Database password: (the one you set in compose file)
   - Database name: `nextcloud`
   - Database host: `nextcloud-db`
5. Click **Install**
6. Wait for installation (may take a few minutes)

#### Functionality Test 2.3C: Verify Nextcloud Functionality

1. After installation, log in to Nextcloud
2. Test file upload: Upload a test file
3. Test file download: Download the test file
4. Navigate to: **Settings** → **Overview**
5. Check for warnings/errors

**Expected Results:**
- Can upload files successfully
- Can download files
- No critical errors in Overview page
- Redis caching working (check Overview for confirmation)

#### Functionality Test 2.3D: Configure Trusted Domains

```bash
# Add tower-of-joy to trusted domains
docker exec -u www-data nextcloud php occ config:system:set trusted_domains 1 --value=tower-of-joy

# If using IP address, add it too
docker exec -u www-data nextcloud php occ config:system:set trusted_domains 2 --value=192.168.x.x
```

**Test:** Access Nextcloud using hostname/IP to verify trusted domains

✅ **Checkpoint 2.3:** Nextcloud deployed and functional with database and Redis

---

### Step 2.4: Deploy Samba File Server

Deploy Samba for Windows/Mac network file sharing.

#### Create Stack in Portainer

1. Navigate to: **Stacks** → **Add Stack**
2. Name: `samba`
3. Web editor: Paste the following configuration

#### Docker Compose Configuration

```yaml
version: '3.8'

services:
  samba:
    image: dperson/samba
    container_name: samba
    restart: unless-stopped
    ports:
      - "139:139"
      - "445:445"
    environment:
      - TZ=Europe/Amsterdam
      - USERID=1000  # Your user ID (run `id -u`)
      - GROUPID=1000 # Your group ID (run `id -g`)
    volumes:
      - /home/jpmschweitzer/docker-data/samba:/share/config  # Config on SSD
      - /mnt/media/jellyfin:/share/media  # Media from 4TB HDD
      - /mnt/media/nextcloud/data:/share/nextcloud:ro  # Nextcloud data from 4TB HDD (read-only)
      - /mnt/media/downloads:/share/downloads  # Downloads folder from 4TB HDD
    command: '-s "Media;/share/media;yes;no;yes;all" -s "Nextcloud;/share/nextcloud;yes;no;yes;all" -s "Downloads;/share/downloads;yes;no;no;all" -u "jpmschweitzer;changeme_samba_password" -p'
```

**Note:** Replace `changeme_samba_password` with a strong password

4. Click **Deploy the stack**

#### Functionality Test 2.4A: Verify Samba Running

```bash
# Check Samba container
docker ps | grep samba

# Check Samba logs
docker logs samba
```

**Expected Results:**
- Container running
- SMB daemon started
- Shares configured

#### Functionality Test 2.4B: Test Network Share Access

**From Windows:**
1. Open File Explorer
2. Type in address bar: `\\tower-of-joy\Media`
3. Enter credentials: `jpmschweitzer` / (your password)
4. Verify you can browse files

**From Mac:**
1. Finder → Go → Connect to Server
2. Enter: `smb://tower-of-joy/Media`
3. Enter credentials
4. Verify access

**From Linux:**
```bash
# Install smbclient if needed
sudo apt-get install smbclient

# Test connection
smbclient -L tower-of-joy -U jpmschweitzer
```

**Expected Results:**
- Can connect to shares
- Can browse files
- Can read/write (depending on share permissions)

✅ **Checkpoint 2.4:** Samba file sharing working from network devices

---

## Phase 2 Completion Checklist

- [ ] Directory structure created
- [ ] Jellyfin deployed and accessible at port 8096
- [ ] Jellyfin GPU transcoding working
- [ ] Nextcloud deployed and accessible at port 8080
- [ ] Nextcloud database and Redis running
- [ ] Nextcloud file upload/download working
- [ ] Samba file server deployed
- [ ] Network shares accessible from other devices

**Phase 2 Status:** Core services operational

---

*For questions or issues, refer to troubleshooting guide or consult service documentation*