portainer-core/plans/active/security-implementation-plan.md

# Security Implementation Plan v2.0: Google OAuth SSO for Homelab Infrastructure

**Version:** 2.0 (Post-Failure Analysis & Comprehensive Revision)
**Date:** 2025-11-20
**Status:** Ready for Implementation
**Previous Version:** 1.1 (Failed - Redirect loops and high memory usage)

## Document Change Log

| Version | Date | Changes | Author |
|---------|------|---------|--------|
| 1.0 | 2025-11-10 | Initial plan | - |
| 1.1 | 2025-11-15 | Scope revision | - |
| 2.0 | 2025-11-20 | Post-failure analysis, redirect loop fixes, memory optimization, milestone checkpoints | Claude Code |

---

## Executive Summary

This document is a **complete revision** of the Authentik SSO implementation plan after a failed first attempt that resulted in infinite redirect loops and excessive memory usage (20-30% of system RAM).

**What Went Wrong (v1.0 Deployment):**
- ✗ Infinite redirect loops between NPM, Authentik, and Google OAuth
- ✗ `auth.schweitz.net` was protected by forward auth, creating self-referential loop
- ✗ Mixed hostname/IP/domain addressing caused OAuth callback failures
- ✗ Missing `AUTHENTIK_HOST` environment variable led to incorrect internal redirects
- ✗ Cookie domain not configured, preventing cross-service SSO
- ✗ Excessive memory usage (~3-5GB) due to separate PostgreSQL and Redis instances
- ✗ No rollback checkpoints, making recovery difficult

**What's Different in v2.0:**
- ✅ Detailed redirect loop prevention strategies
- ✅ Strict hostname/URL consistency requirements
- ✅ Memory optimization with shared PostgreSQL/Redis (target: <1GB total)
- ✅ **Milestone-based deployment with rollback points**
- ✅ **Configuration versioning at each step**
- ✅ **Validation tests before proceeding to next milestone**
- ✅ **Progress tracking within this document**
- ✅ Embedded outpost instead of separate container (simpler)

**Critical Success Factors:**
1. `auth.schweitz.net` must **NEVER** have forward auth enabled
2. `AUTHENTIK_HOST=https://auth.schweitz.net` must be set correctly
3. `AUTHENTIK_COOKIE_DOMAIN=.schweitz.net` for cross-service SSO
4. Test each service individually before moving to the next
5. Create backup snapshots at every milestone
6. Update this document with progress and issues as we go

**SSO Inclusion Policy:**
- ✅ **Include:** Web-based admin interfaces, dashboards, APIs requiring browser access
- ❌ **Exclude:** Services with native mobile/desktop apps that work better with username/password
- ❌ **Exclude:** Media streaming services (Jellyfin) - app integration priority
- ❌ **Exclude:** Development tools (code-server) - IDE integration priority
- ⏸️ **Defer:** Disabled/inactive services (Nextcloud) - implement when re-enabled

---

## Table of Contents

- [Section 0: Failure Analysis & Lessons Learned](#section-0-failure-analysis--lessons-learned)
- [Section 1: Prerequisites & Environment Validation](#section-1-prerequisites--environment-validation)
- [Section 2: Resource Optimization Strategy](#section-2-resource-optimization-strategy)
- [Section 3: Redirect Loop Prevention](#section-3-redirect-loop-prevention)
- [Section 4: Implementation Milestones](#section-4-implementation-milestones)
- [Section 5: Testing & Validation](#section-5-testing--validation)
- [Section 6: Troubleshooting Guide](#section-6-troubleshooting-guide)
- [Section 7: Rollback Procedures](#section-7-rollback-procedures)
- [Section 8: Progress Tracking](#section-8-progress-tracking)

---

## Section 0: Failure Analysis & Lessons Learned

### 0.1 What Happened in v1.0 Deployment

**Timeline:**
- Deployed Authentik stack with separate PostgreSQL and Redis
- Configured forward auth on ALL NPM proxy hosts (including auth.schweitz.net)
- Attempted to access services → immediate redirect loops
- Consumed 20-30% of system RAM (~3-5GB)
- Rolled back to working state (commit: `cb428a8`)

**Root Causes Identified:**

#### Issue 1: Self-Referential Authentication Loop
```
User → https://api.schweitz.net
  → NPM: "Need auth, redirect to https://auth.schweitz.net"
  → https://auth.schweitz.net
  → NPM: "Need auth, redirect to https://auth.schweitz.net"  ← LOOP!
```

**Why it happened:** NPM configuration had forward auth enabled on `auth.schweitz.net` itself.

**Fix:** Remove forward auth from `auth.schweitz.net` NPM proxy host configuration.

#### Issue 2: Hostname/URL Discrepancies
```
External: https://auth.schweitz.net/source/oauth/callback/google/
Internal: http://localhost:9000/source/oauth/callback/google/
Outpost:  http://192.168.86.149:9001/...
```

**Why it happened:**
- Mixed use of localhost, IP addresses, and domain names
- `AUTHENTIK_HOST` environment variable not set
- Authentik didn't know its external URL

**Fix:**
- Set `AUTHENTIK_HOST=https://auth.schweitz.net`
- Use consistent container names (not localhost/IP) in Docker configs
- Use external URLs for all OAuth callbacks

#### Issue 3: Cookie Domain Mismatch
**Why it happened:** Authentik didn't set cookie domain, so cookies were scoped to individual hosts, not shared across `*.schweitz.net`.

**Fix:** Set `AUTHENTIK_COOKIE_DOMAIN=.schweitz.net`

#### Issue 4: Excessive Memory Usage
**Why it happened:**
- Separate PostgreSQL instance for Authentik (~1GB)
- Separate Redis instance for Authentik (~100MB)
- No resource limits on server/worker containers

**Fix:**
- Use shared PostgreSQL (`postgres-shared:5432`)
- Use shared Redis (`redis-shared:6379` DB 0)
- Set memory limits: 512M server, 384M worker

### 0.2 Evidence from NPM Backup (2025-11-19)

All 11 NPM proxy hosts had **identical forward auth configuration**:
```nginx
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @authentik_proxy_signin;

location @authentik_proxy_signin {
    return 302 /outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
}

location /outpost.goauthentik.io {
    proxy_pass http://192.168.86.149:9001/outpost.goauthentik.io;
}
```

**Problem:** This configuration was applied to ALL hosts, including `auth.schweitz.net`.

### 0.3 Lessons Learned

1. **Never protect the authentication server with authentication** - Seems obvious in retrospect, but easy to miss
2. **Test incrementally** - We tried to enable all 11 services at once
3. **Document assumptions** - We assumed Authentik would auto-detect external URLs
4. **Create rollback points** - Had to manually restore NPM config from backup
5. **Monitor resource usage** - Didn't realize memory consumption until it was running
6. **Read the docs thoroughly** - `AUTHENTIK_HOST` is a critical but easy-to-miss setting

---

## Section 1: Prerequisites & Environment Validation

### 1.1 System Specifications

| Component | Specification | Notes |
|-----------|---------------|-------|
| CPU | Intel i7-6700 (4C/8T) | ~19% usage for Authentik (1.5 cores) |
| RAM | 16GB total | Target: <1GB for Authentik (<6.25% of total) |
| Storage (SSD) | 489GB | ~3GB needed for Authentik data |
| Storage (HDD) | 3.7TB | Not used by Authentik |
| Network | docker-dataplane | All services on unified network |

**Available Resources:**
- RAM: 16GB - Current usage = ~6-8GB free
- Authentik target: 896MB (5.6% of total RAM) ✓ Acceptable

### 1.2 Shared Infrastructure Status

**PostgreSQL Shared** (`postgres-shared:5432`):
```bash
# Verify connectivity
docker exec postgres-shared pg_isready

# Verify authentik database exists
docker exec postgres-shared psql -U postgres -c "\l" | grep authentik

# Test connection as authentik_user
docker exec postgres-shared psql -U authentik_user -d authentik -c "SELECT version();"
```

**Expected output:**
```
authentik | authentik_user | UTF8 | ...
PostgreSQL 16.x on x86_64-pc-linux-musl...
```

**Redis Shared** (`redis-shared:6379`):
```bash
# Verify connectivity
docker exec redis-shared redis-cli PING

# Check DB 0 is available (Authentik allocation)
docker exec redis-shared redis-cli -n 0 INFO keyspace

# Test write/read
docker exec redis-shared redis-cli -n 0 SET test:authentik "connection_ok"
docker exec redis-shared redis-cli -n 0 GET test:authentik
docker exec redis-shared redis-cli -n 0 DEL test:authentik
```

**Expected output:**
```
PONG
# Keyspace (should be empty or have other keys)
connection_ok
1
```

### 1.3 Network Verification

```bash
# Verify docker-dataplane network exists
docker network inspect docker-dataplane

# List all containers on docker-dataplane
docker network inspect docker-dataplane | grep -A 2 "Containers"

# Verify DNS resolution between containers
docker run --rm --network docker-dataplane alpine ping -c 3 postgres-shared
docker run --rm --network docker-dataplane alpine ping -c 3 redis-shared
```

**Expected:** 19 containers on network, postgres-shared and redis-shared resolve correctly.

### 1.4 NPM Current State

```bash
# Backup current NPM configuration
curl -s http://192.168.86.149:8000/api/ > backups/npm-pre-authentik-$(date +%Y%m%d-%H%M%S).json

# Verify auth.schweitz.net proxy host exists
cat backups/npm-pre-authentik-*.json | jq '.proxy_hosts[] | select(.domain_names[] | contains("auth.schweitz"))'
```

**Expected:** auth.schweitz.net points to localhost:9000, currently has forward auth (will be removed).

### 1.5 SSL Certificates

```bash
# Verify Let's Encrypt certificates for schweitz.net domains
docker exec npm ls -la /etc/letsencrypt/live/ | grep schweitz
```

**Expected:** Certificates for all *.schweitz.net domains, valid and auto-renewing.

---

## Section 2: Resource Optimization Strategy

### 2.1 Resource Allocation Plan

**Authentik Server:**
```yaml
deploy:
  resources:
    limits:
      memory: 512M       # Hard cap
      cpus: '0.5'        # 50% of one core
    reservations:
      memory: 256M       # Guaranteed minimum
```

**Authentik Worker:**
```yaml
deploy:
  resources:
    limits:
      memory: 384M       # Hard cap
      cpus: '0.3'        # 30% of one core
    reservations:
      memory: 128M       # Guaranteed minimum
```

**Total Authentik overhead:** 896MB (vs previous 3-5GB = **78% reduction**)

### 2.2 Worker Configuration

**Reduce Celery workers and threads:**
```yaml
environment:
  # Celery worker configuration
  AUTHENTIK_BOOTSTRAP_WORKERS: 1          # Default: 1 (single worker)
  AUTHENTIK_WORKER__CONCURRENCY: 2        # Threads per worker (default: 2)

  # Logging optimization
  AUTHENTIK_LOG_LEVEL: warning            # Reduce log verbosity
  AUTHENTIK_ERROR_REPORTING__ENABLED: false  # Disable error reporting

  # Disable non-essential features
  AUTHENTIK_AVATARS: none                 # Disable avatar fetching (saves bandwidth)
  AUTHENTIK_FOOTER_LINKS: '[]'           # Disable footer links
```

**Resource impact:**
- 1 worker @ 2 threads = minimal CPU usage during idle
- Warning-level logging = less I/O overhead
- No avatar fetching = no external HTTP requests

### 2.3 Database Connection Optimization

**Use shared infrastructure, no connection pooling (unnecessary for homelab scale):**
```yaml
environment:
  # PostgreSQL connection
  AUTHENTIK_POSTGRESQL__HOST: postgres-shared
  AUTHENTIK_POSTGRESQL__PORT: 5432
  AUTHENTIK_POSTGRESQL__NAME: authentik
  AUTHENTIK_POSTGRESQL__USER: authentik_user
  AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_DB_PASSWORD}
  AUTHENTIK_POSTGRESQL__USE_PGBOUNCER: false     # No pooler needed
  AUTHENTIK_POSTGRESQL__USE_PGPOOL: false        # No pooler needed

  # Redis connection
  AUTHENTIK_REDIS__HOST: redis-shared
  AUTHENTIK_REDIS__PORT: 6379
  AUTHENTIK_REDIS__DB: 0                         # Dedicated DB for Authentik
  AUTHENTIK_REDIS__PASSWORD: ""                  # No password (internal network only)
```

**Benefit:** No dedicated PostgreSQL/Redis = ~1.1GB RAM saved.

### 2.4 Memory Usage Monitoring

**Check memory usage after deployment:**
```bash
# Container-level memory usage
docker stats authentik-server authentik-worker --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Process-level memory inside containers
docker exec authentik-server ps aux --sort=-%mem | head -10
docker exec authentik-worker ps aux --sort=-%mem | head -10
```

**Expected values:**
- authentik-server: 200-400MB (under 512M limit)
- authentik-worker: 100-250MB (under 384M limit)
- **Total: 300-650MB** (healthy, with burst headroom)

**If memory exceeds limits:**
- Check logs for errors: `docker logs authentik-server --tail 100`
- Verify no memory leaks: `docker exec authentik-server free -h`
- Restart containers if needed: `docker restart authentik-server authentik-worker`

---

## Section 3: Redirect Loop Prevention

### 3.1 Critical Configuration Rules

**RULE #1: Never Enable Forward Auth on auth.schweitz.net**

```nginx
# ✓ CORRECT - auth.schweitz.net NPM config
{
  "domain_names": ["auth.schweitz.net"],
  "forward_host": "localhost",
  "forward_port": 9000,
  "ssl_forced": true,
  "advanced_config": "# NO FORWARD AUTH - This is the login page!\n\n# Optional: Rate limiting\nlimit_req_zone $binary_remote_addr zone=auth_limit:10m rate=10r/m;\nlimit_req zone=auth_limit burst=5 nodelay;"
}
```

```nginx
# ✗ WRONG - DO NOT DO THIS!
{
  "domain_names": ["auth.schweitz.net"],
  "advanced_config": "auth_request /outpost.goauthentik.io/auth/nginx;"  # ← CAUSES LOOP!
}
```

**RULE #2: Set AUTHENTIK_HOST to External URL**

```yaml
environment:
  AUTHENTIK_HOST: https://auth.schweitz.net              # ← External HTTPS URL
  AUTHENTIK_HOST_BROWSER: https://auth.schweitz.net      # ← Same as AUTHENTIK_HOST
```

**Why:** Authentik uses this for generating redirect URLs in OAuth flows. If unset, it uses the request Host header, which might be `localhost` or an IP address internally.

**RULE #3: Configure Cookie Domain for SSO**

```yaml
environment:
  AUTHENTIK_COOKIE_DOMAIN: .schweitz.net                 # ← Note the leading dot!
  AUTHENTIK_COOKIE_SAMESITE: lax                         # ← Allow cross-subdomain cookies
```

**Why:** Without this, cookies are scoped to individual hosts (e.g., `auth.schweitz.net` only), and can't be shared with `api.schweitz.net`, `cloud.schweitz.net`, etc.

**RULE #4: Use Embedded Outpost (Simplicity)**

```yaml
# Server container serves both web UI and outpost
authentik-server:
  ports:
    - "9000:9000"   # Web UI
    - "9443:9443"   # Embedded outpost
  environment:
    AUTHENTIK_OUTPOSTS__DOCKER_IMAGE_BASE: "ghcr.io/goauthentik/%(type)s:%(version)s"
```

**Why:** Eliminates need for separate outpost container, reduces complexity and memory usage.

**RULE #5: Consistent NPM Forward Auth Configuration**

```nginx
# Forward auth configuration for protected services (NOT auth.schweitz.net)
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @authentik_proxy_signin;

auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;

auth_request_set $authentik_username $upstream_http_x_authentik_username;
auth_request_set $authentik_email $upstream_http_x_authentik_email;
auth_request_set $authentik_groups $upstream_http_x_authentik_groups;

proxy_set_header X-authentik-username $authentik_username;
proxy_set_header X-authentik-email $authentik_email;
proxy_set_header X-authentik-groups $authentik_groups;

location @authentik_proxy_signin {
    internal;
    return 302 https://auth.schweitz.net/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
}

location /outpost.goauthentik.io {
    proxy_pass http://authentik-server:9443/outpost.goauthentik.io;  # ← Use embedded outpost
    proxy_set_header Host $host;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
}
```

### 3.2 OAuth Redirect URI Configuration

**Google OAuth Callback URL:** `https://auth.schweitz.net/source/oauth/callback/google/`

**In Google Cloud Console:**
```
Authorized JavaScript origins:
  https://auth.schweitz.net

Authorized redirect URIs:
  https://auth.schweitz.net/source/oauth/callback/google/
```

**In Authentik (Google Social Source):**
```
Consumer Key: <Google Client ID>
Consumer Secret: <Google Client Secret>
Provider Type: Google
Callback URL: (auto-filled, verify it matches Google Cloud Console)
```

**Critical:** The callback URL must **exactly** match between Google Cloud Console and Authentik. Case-sensitive, trailing slashes matter.

### 3.3 Testing Redirect Flows

**Manual redirect flow test:**
```bash
# 1. Visit protected service (should redirect to Authentik)
curl -I https://api.schweitz.net
# Expected: 302 redirect to https://auth.schweitz.net/outpost.goauthentik.io/start?rd=...

# 2. Visit Authentik login page (should NOT redirect)
curl -I https://auth.schweitz.net
# Expected: 200 OK (Authentik login page)

# 3. After login, check if cookie is set
curl -I -c cookies.txt https://auth.schweitz.net/flows/executor/default-provider-authentication-flow/
# Expected: Set-Cookie: authentik_session=...; Domain=.schweitz.net

# 4. Use cookie to access protected service
curl -I -b cookies.txt https://api.schweitz.net
# Expected: 200 OK (authenticated)
```

### 3.4 Redirect Loop Detection

**Symptoms:**
- Browser error: "ERR_TOO_MANY_REDIRECTS"
- Nginx logs show repeated 302 responses
- Authentik logs show repeated authentication attempts from same IP

**Diagnosis:**
```bash
# Check NPM logs for redirect chains
docker logs npm 2>&1 | grep -A 5 "auth.schweitz.net" | tail -50

# Check Authentik logs for repeated auth attempts
docker logs authentik-server 2>&1 | grep -i "redirect\|loop\|401\|302" | tail -50

# Test with verbose curl (shows redirect chain)
curl -v -L https://api.schweitz.net 2>&1 | grep -E "(< HTTP|< Location)"
```

**If loop detected:**
1. Check if `auth.schweitz.net` has forward auth enabled → **Remove it**
2. Verify `AUTHENTIK_HOST` environment variable → Should be `https://auth.schweitz.net`
3. Check cookie domain → Should be `.schweitz.net`
4. Verify outpost endpoint is reachable → `curl http://authentik-server:9443/outpost.goauthentik.io/ping/`

---

## Section 4: Implementation Milestones

### Milestone 0: Pre-Deployment Validation ✓ Safe to Execute

**Objective:** Verify all prerequisites are met before deploying Authentik.

**Tasks:**
- [ ] **M0.1:** Run PostgreSQL connectivity tests (Section 1.2)
- [ ] **M0.2:** Run Redis connectivity tests (Section 1.2)
- [ ] **M0.3:** Verify docker-dataplane network (Section 1.3)
- [ ] **M0.4:** Backup NPM configuration
- [ ] **M0.5:** Document current NPM forward auth status (should have auth on all hosts)
- [ ] **M0.6:** Remove forward auth from ALL NPM proxy hosts temporarily
- [ ] **M0.7:** Verify system resource availability (RAM, CPU, disk)

**Validation Checklist:**
```bash
# Run all validation commands
bash scripts/validate-authentik-prerequisites.sh  # Create this script
```

**Expected Duration:** 15-30 minutes

**Rollback:** N/A (no changes made to running services)

**Backup Files:**
- `backups/npm-m0-$(date +%Y%m%d).json` - NPM config before changes
- `backups/docker-ps-m0-$(date +%Y%m%d).txt` - Container list

**Status:** ⏳ Not Started

**Issues Log:**
```
[Date] [Issue description] - [Resolution]
```

---

### Milestone 1: Authentik Deployment (Core Only)

**Objective:** Deploy Authentik server and worker containers, configure NPM proxy for auth.schweitz.net (WITHOUT forward auth).

**Dependencies:** M0 completed

**Tasks:**
- [ ] **M1.1:** Create `stacks/authentik.yml` with correct environment variables
- [ ] **M1.2:** Set AUTHENTIK_HOST, AUTHENTIK_COOKIE_DOMAIN, resource limits
- [ ] **M1.3:** Deploy Authentik stack via Portainer
- [ ] **M1.4:** Wait for containers to start (watch logs)
- [ ] **M1.5:** Verify database migrations completed
- [ ] **M1.6:** Update NPM proxy host for `auth.schweitz.net` (NO forward auth!)
- [ ] **M1.7:** Access https://auth.schweitz.net (should show Authentik setup wizard)
- [ ] **M1.8:** Complete initial setup wizard, create admin account

**Configuration Files:**

**`stacks/authentik.yml`:**
```yaml
version: '3.8'

# Authentik Identity Provider (SSO)
# Purpose: Centralized authentication for all homelab services
# Ports: 9000 (web UI), 9443 (embedded outpost)
# GPU: No
# Storage: SSD (configs), PostgreSQL shared (user data)

services:
  authentik-server:
    image: ghcr.io/goauthentik/server:2024.8.4  # Pinned version (2024.10 has redirect loop issues)
    container_name: authentik-server
    restart: unless-stopped
    command: server
    environment:
      # External URLs (CRITICAL for redirect loop prevention)
      AUTHENTIK_HOST: https://auth.schweitz.net
      AUTHENTIK_HOST_BROWSER: https://auth.schweitz.net

      # Cookie settings (CRITICAL for SSO across subdomains)
      AUTHENTIK_COOKIE_DOMAIN: .schweitz.net
      AUTHENTIK_COOKIE_SAMESITE: lax

      # SSL/TLS
      AUTHENTIK_INSECURE: false

      # PostgreSQL (shared)
      AUTHENTIK_POSTGRESQL__HOST: postgres-shared
      AUTHENTIK_POSTGRESQL__PORT: 5432
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__USER: authentik_user
      AUTHENTIK_POSTGRESQL__PASSWORD: F//j0ktck7cX06Vfgh0YXceONOtlSsHvadqROICeDx8=
      AUTHENTIK_POSTGRESQL__USE_PGBOUNCER: false

      # Redis (shared)
      AUTHENTIK_REDIS__HOST: redis-shared
      AUTHENTIK_REDIS__PORT: 6379
      AUTHENTIK_REDIS__DB: 0

      # Secret key (generate with: openssl rand -base64 32)
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}  # Store in .env file

      # Resource optimization
      AUTHENTIK_LOG_LEVEL: warning
      AUTHENTIK_ERROR_REPORTING__ENABLED: false
      AUTHENTIK_AVATARS: none
      AUTHENTIK_FOOTER_LINKS: '[]'

      # Embedded outpost configuration
      AUTHENTIK_OUTPOSTS__DOCKER_IMAGE_BASE: "ghcr.io/goauthentik/%(type)s:%(version)s"

      # Timezone
      TZ: Europe/Amsterdam

    ports:
      - "9000:9000"   # Web UI
      - "9443:9443"   # Embedded outpost (proxy provider)

    volumes:
      - /home/jpmschweitzer/docker-data/authentik/media:/media
      - /home/jpmschweitzer/docker-data/authentik/custom-templates:/templates

    networks:
      - docker-dataplane

    depends_on:
      - postgres-shared
      - redis-shared

    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:9000/-/health/live/ || exit 1"]
      start_period: 60s
      interval: 30s
      timeout: 10s
      retries: 3

    deploy:
      resources:
        limits:
          memory: 512M
          cpus: '0.5'
        reservations:
          memory: 256M

  authentik-worker:
    image: ghcr.io/goauthentik/server:2024.8.4  # Same version as server
    container_name: authentik-worker
    restart: unless-stopped
    command: worker
    environment:
      # Same environment as server (MUST match exactly)
      AUTHENTIK_HOST: https://auth.schweitz.net
      AUTHENTIK_HOST_BROWSER: https://auth.schweitz.net
      AUTHENTIK_COOKIE_DOMAIN: .schweitz.net
      AUTHENTIK_COOKIE_SAMESITE: lax
      AUTHENTIK_INSECURE: false
      AUTHENTIK_POSTGRESQL__HOST: postgres-shared
      AUTHENTIK_POSTGRESQL__PORT: 5432
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__USER: authentik_user
      AUTHENTIK_POSTGRESQL__PASSWORD: F//j0ktck7cX06Vfgh0YXceONOtlSsHvadqROICeDx8=
      AUTHENTIK_POSTGRESQL__USE_PGBOUNCER: false
      AUTHENTIK_REDIS__HOST: redis-shared
      AUTHENTIK_REDIS__PORT: 6379
      AUTHENTIK_REDIS__DB: 0
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_LOG_LEVEL: warning
      AUTHENTIK_ERROR_REPORTING__ENABLED: false
      TZ: Europe/Amsterdam

      # Worker-specific configuration
      AUTHENTIK_BOOTSTRAP_WORKERS: 1   # Single worker (homelab scale)
      AUTHENTIK_WORKER__CONCURRENCY: 2  # 2 threads per worker

    volumes:
      - /home/jpmschweitzer/docker-data/authentik/media:/media
      - /home/jpmschweitzer/docker-data/authentik/custom-templates:/templates
      - /home/jpmschweitzer/docker-data/authentik/certs:/certs
      - /var/run/docker.sock:/var/run/docker.sock  # For outpost management

    networks:
      - docker-dataplane

    depends_on:
      - postgres-shared
      - redis-shared
      - authentik-server

    healthcheck:
      test: ["CMD-SHELL", "ak healthcheck"]
      start_period: 60s
      interval: 30s
      timeout: 10s
      retries: 3

    deploy:
      resources:
        limits:
          memory: 384M
          cpus: '0.3'
        reservations:
          memory: 128M

networks:
  docker-dataplane:
    external: true
    name: docker-dataplane

# Setup Instructions:
#
# 1. Generate secret key:
#    openssl rand -base64 32 > .env.authentik
#    echo "AUTHENTIK_SECRET_KEY=<paste-key-here>" >> .env.authentik
#
# 2. Create directories:
#    mkdir -p ~/docker-data/authentik/{media,custom-templates,certs}
#
# 3. Deploy stack:
#    docker-compose -f stacks/authentik.yml up -d
#
# 4. Watch logs:
#    docker logs -f authentik-server
#    docker logs -f authentik-worker
#
# 5. Wait for migrations to complete (~2-3 minutes):
#    docker logs authentik-server 2>&1 | grep "Applying migration"
#
# 6. Access web UI:
#    https://auth.schweitz.net (should show setup wizard)
#
# 7. Complete setup wizard:
#    - Email: admin@schweitz.net
#    - Password: <secure-password>
#    - Finish setup
#
# Monitoring:
#
# Memory usage:
#   docker stats authentik-server authentik-worker --no-stream
#
# Database connectivity:
#   docker exec authentik-server ak check
#
# Outpost status:
#   curl http://authentik-server:9443/outpost.goauthentik.io/ping/
```

**Validation Commands:**
```bash
# M1.1: Check containers are running
docker ps | grep authentik

# M1.2: Check logs for errors
docker logs authentik-server 2>&1 | grep -iE "error|critical|fail" | tail -20
docker logs authentik-worker 2>&1 | grep -iE "error|critical|fail" | tail -20

# M1.3: Check database migrations completed
docker logs authentik-server 2>&1 | grep "Applying migration" | tail -10
docker logs authentik-server 2>&1 | grep "No migrations to apply" || echo "❌ Migrations still running"

# M1.4: Check health checks passing
docker inspect authentik-server | jq '.[0].State.Health.Status'
docker inspect authentik-worker | jq '.[0].State.Health.Status'

# M1.5: Check memory usage (should be under limits)
docker stats authentik-server authentik-worker --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"

# M1.6: Check NPM proxy host for auth.schweitz.net
curl -I https://auth.schweitz.net
# Expected: 200 OK (Authentik setup wizard)

# M1.7: Check for redirect loops (should NOT redirect)
curl -v https://auth.schweitz.net 2>&1 | grep -c "Location:"
# Expected: 0 (or 1 if redirecting HTTP → HTTPS, which is fine)

# M1.8: Verify embedded outpost is accessible
docker exec -it authentik-server curl http://localhost:9443/outpost.goauthentik.io/ping/
# Expected: {"status": "ok"} or similar
```

**Success Criteria:**
- ✓ Both containers running (docker ps)
- ✓ Health checks passing (docker inspect)
- ✓ Memory usage < 700MB total
- ✓ Database migrations completed
- ✓ https://auth.schweitz.net accessible (200 OK, no redirect loop)
- ✓ Authentik setup wizard displayed
- ✓ Admin account created

**Expected Duration:** 30-60 minutes

**Rollback Procedure:**
```bash
# Stop and remove Authentik containers
docker-compose -f stacks/authentik.yml down

# Restore NPM configuration (if changed)
curl -X POST http://192.168.86.149:8000/api/restore \
  -H "Content-Type: application/json" \
  -d @backups/npm-m0-YYYYMMDD.json

# Verify NPM restored
curl -I https://auth.schweitz.net  # Should return error (Authentik not running)
```

**Backup Files:**
- `backups/authentik-m1-$(date +%Y%m%d).yml` - Authentik stack config
- `backups/npm-m1-$(date +%Y%m%d).json` - NPM config after M1
- `backups/authentik-m1-env-$(date +%Y%m%d).txt` - Environment variables (REDACTED)

**Status:** ⏳ Not Started

**Issues Log:**
```
[Date] [Issue description] - [Resolution]
```

---

### Milestone 2: Google OAuth Integration

**Objective:** Configure Google as an authentication source in Authentik.

**Dependencies:** M1 completed successfully

**Tasks:**
- [ ] **M2.1:** Create Google Cloud Project (or use existing)
- [ ] **M2.2:** Configure OAuth consent screen
- [ ] **M2.3:** Create OAuth 2.0 client credentials
- [ ] **M2.4:** Add Google as social source in Authentik admin panel
- [ ] **M2.5:** Configure callback URLs
- [ ] **M2.6:** Test Google login with Workspace account
- [ ] **M2.7:** Test Google login with Gmail account
- [ ] **M2.8:** Verify user profile attributes synced

**Configuration:** See original plan Section 5.1.2 for detailed steps.

**Expected Duration:** 30-45 minutes

**Status:** ⏳ Not Started

---

### Milestone 3: Single Service Protection (Organizr Test)

**Objective:** Enable forward auth on ONE service (Organizr at home.schweitz.net) to validate SSO works end-to-end.

**Dependencies:** M2 completed successfully

**Configuration:** See original plan Section 5.2 for detailed steps.

**Expected Duration:** 60-90 minutes

**Status:** ⏳ Not Started

---

### Milestone 4: Core API Protection (FastAPI Native OIDC)

**Objective:** Protect Core API with FastAPI native OIDC token validation (bearer token authentication).

**Dependencies:** M3 completed successfully

**Status:** ✅ **COMPLETE** - Using forward auth (shares Organizr Proxy provider)

**Implementation Decision (2025-11-23):**
- Core API already protected with forward auth via NPM
- Shares "Organizr Proxy" provider with home.schweitz.net
- Authentication working correctly with Google OAuth
- Headers forwarded: X-authentik-username, X-authentik-email, X-authentik-groups, X-authentik-name, X-authentik-uid
- **Decision:** Keep current setup, defer separate admin provider to avoid complexity
- **Rationale:** Current implementation is secure and functional for homelab use case

**Configuration:**
- Provider: Organizr Proxy (shared)
- External host: https://api.schweitz.net
- Outpost: Standalone proxy (port 9445)
- Mode: forward_single

**Expected Duration:** ~~90-120 minutes~~ SKIPPED (already functional)

---

### Milestone 5: Remaining Services (Gradual Rollout)

**Objective:** Enable forward auth on remaining services, one at a time, testing each before proceeding.

**Dependencies:** M3 and M4 completed successfully

**Services to Protect:**
- Gitea (git.schweitz.net)
- Open WebUI (no external domain yet)
- Netdata (no external domain yet)
- Uptime Kuma (no external domain yet)
- AMP (amp.schweitz.net)
- Tatlock (tatlock.schweitz.net)

**Services EXCLUDED from SSO (Keep Native Auth):**
- ❌ **Jellyfin (media.schweitz.net)** - Better mobile app integration with native auth
- ❌ **code-server (code.schweitz.net)** - Better VS Code integration with native auth
- ❌ **Nextcloud (cloud.schweitz.net)** - Service disabled, SSO deferred until re-enabled

**Rationale for Exclusions:**
- Jellyfin and code-server have excellent native authentication
- Mobile apps and desktop clients work better with username/password
- SSO adds complexity without significant security benefit for these services
- Nextcloud is not currently in active use

**Expected Duration:** 3-6 hours (30-60 min per service)

**Status:** ⏳ Not Started

---

## Section 5: Testing & Validation

### 5.1 Automated Testing Scripts

**Create these scripts in `/scripts` directory:**

- `validate-authentik-prerequisites.sh` - Pre-deployment checks
- `test-redirect-flow.sh` - Test redirect loops
- `monitor-resources.sh` - Monitor memory/CPU usage

(Full script contents in original plan appendices)

---

## Section 6: Troubleshooting Guide

### 6.1 Redirect Loop Troubleshooting

**Common Causes & Fixes:**

| Cause | Fix |
|-------|-----|
| `auth.schweitz.net` has forward auth enabled | Remove forward auth from NPM config |
| `AUTHENTIK_HOST` not set or wrong | Set `AUTHENTIK_HOST=https://auth.schweitz.net` |
| Cookie domain mismatch | Set `AUTHENTIK_COOKIE_DOMAIN=.schweitz.net` |
| Outpost endpoint unreachable | Verify `curl http://authentik-server:9443/outpost.goauthentik.io/ping/` works |

### 6.2 Memory Usage Issues

**Common Causes & Fixes:**

| Cause | Fix |
|-------|-----|
| Too many Celery workers | Set `AUTHENTIK_BOOTSTRAP_WORKERS=1` |
| Too many threads per worker | Set `AUTHENTIK_WORKER__CONCURRENCY=2` |
| Memory leak | Restart containers |

(Full troubleshooting guide in original plan Section 6)

---

## Section 7: Rollback Procedures

### 7.1 Full Rollback

```bash
# Stop Authentik
docker-compose -f stacks/authentik.yml down -v

# Restore NPM config
curl -X POST http://192.168.86.149:8000/api/restore \
  -H "Content-Type: application/json" \
  -d @backups/npm-m0-YYYYMMDD.json

# Verify services accessible
curl -I https://api.schweitz.net
```

### 7.2 Partial Rollback (Single Service)

Restore NPM config for specific proxy host from milestone backup.

(Full rollback procedures in original plan Section 7)

---

## Section 8: Progress Tracking

### 8.1 Implementation Status

**Last Updated:** 2025-11-20

| Milestone | Status | Start Date | Complete Date | Duration | Issues |
|-----------|--------|------------|---------------|----------|--------|
| M0: Pre-deployment Validation | ⏳ Not Started | - | - | - | - |
| M1: Authentik Deployment | ⏳ Not Started | - | - | - | - |
| M2: Google OAuth | ⏳ Not Started | - | - | - | - |
| M3: Organizr SSO | ⏳ Not Started | - | - | - | - |
| M4: Core API OIDC | ⏳ Not Started | - | - | - | - |
| M5: Remaining Services | ⏳ Not Started | - | - | - | - |

**Legend:**
- ⏳ Not Started
- 🔄 In Progress
- ✅ Completed
- ⚠️ Issues/Blocked
- ❌ Failed

### 8.2 Detailed Progress Log

**Update this section as you progress through each milestone.**

**Template:**
```
### Milestone X: [Name]

**Start:** [YYYY-MM-DD HH:MM]
**Completed:** [YYYY-MM-DD HH:MM]
**Duration:** [Time]

**Tasks Completed:**
- [x] Task 1
- [x] Task 2

**Issues Encountered:**
1. **Issue:** [Description]
   **Cause:** [Root cause]
   **Solution:** [How fixed]

**Validation Results:**
- ✓ Item 1
- ✓ Item 2

**Backups Created:**
- backups/filename-mX-YYYYMMDD.ext

**Next Steps:**
- Proceed to Milestone X+1
```

### 8.3 Configuration Changelog

**Track all configuration changes:**

| Date | Component | Change | Reason | Rollback |
|------|-----------|--------|--------|----------|
| 2025-11-20 | Plan | Created v2.0 | Post-failure revision | N/A |
| - | - | - | - | - |

### 8.4 Lessons Learned (Real-Time)

**Add lessons as you encounter issues:**

```
[Date] [Milestone] [Lesson]
```

---

## Appendix A: Quick Reference Commands

### A.1 Authentik Management

```bash
# Check status
docker ps | grep authentik
docker logs authentik-server --tail 50

# Restart
docker restart authentik-server authentik-worker

# Check database
docker exec authentik-server ak check

# Run migrations
docker exec authentik-server ak migrate

# Check outpost
curl http://authentik-server:9443/outpost.goauthentik.io/ping/
```

### A.2 NPM Management

```bash
# Backup
curl -s http://192.168.86.149:8000/api/ > backups/npm-$(date +%Y%m%d-%H%M%S).json

# Test config
docker exec npm nginx -t

# Reload
docker exec npm nginx -s reload
```

---

## Appendix B: Environment Variables Reference

**Authentik Critical Variables:**
```bash
AUTHENTIK_HOST=https://auth.schweitz.net
AUTHENTIK_HOST_BROWSER=https://auth.schweitz.net
AUTHENTIK_COOKIE_DOMAIN=.schweitz.net
AUTHENTIK_COOKIE_SAMESITE=lax
AUTHENTIK_POSTGRESQL__HOST=postgres-shared
AUTHENTIK_REDIS__HOST=redis-shared
AUTHENTIK_LOG_LEVEL=warning
AUTHENTIK_BOOTSTRAP_WORKERS=1
AUTHENTIK_WORKER__CONCURRENCY=2
```

---

## Appendix C: Known Issues & Workarounds

### C.1 Authentik 2024.10.x Redirect Loop Bug

**Issue:** Version 2024.10 has redirect loop bug.

**Workaround:** Use version 2024.8.4 (pinned in authentik.yml).

**Reference:** https://github.com/goauthentik/authentik/issues/11883

---

**END OF SECURITY IMPLEMENTATION PLAN v2.0**

**Ready for Implementation:** Yes ✓
**Reviewed By:** Claude Code (Autonomous Analysis)
**Approved By:** [User - Pending]

*This plan is a living document. Update Section 8 (Progress Tracking) as you progress through milestones.*