portainer-core/docs/sessions/2025-11-20-authentik-deployment.md

# Authentik SSO Deployment Session

**Date:** 2025-11-20
**Duration:** ~4 hours
**Status:** Milestone 2/5 Complete (Google OAuth Working)
**Version:** 0.8.0-authentik-sso

## Session Overview

Successfully deployed Authentik identity provider with Google OAuth integration and optimized memory usage. Forward authentication configuration blocked on embedded outpost initialization issue.

---

## Accomplishments

### ✅ Milestone 1: Authentik Deployment (COMPLETE)

**Infrastructure Setup:**
- Deployed Authentik server and worker containers (version 2024.8.4)
- Configured shared PostgreSQL: `authentik` database with `authentik_user`
- Configured shared Redis: Database 0
- Network: Connected to `docker-dataplane`

**Configuration Highlights:**
```yaml
Memory Limits:
  - Server: 512M limit, 256M reservation
  - Worker: 384M limit, 128M reservation
  - Total: 563MB actual usage (vs 3-5GB previous attempt = 80-90% reduction!)

Ports:
  - 9000: Web UI
  - 9444: Embedded outpost (mapped from container 9443)

Environment:
  - AUTHENTIK_HOST: https://auth.schweitz.net
  - AUTHENTIK_COOKIE_DOMAIN: .schweitz.net
  - PostgreSQL: postgres-shared:5432/authentik
  - Redis: redis-shared:6379/0
```

**Issues Resolved:**
1. **Health check failure** - Container didn't have wget/curl
   - Solution: Used Python's urllib.request for health checks
2. **Database user didn't exist** - authentik_user not created by init script
   - Solution: Manually created user with proper grants
3. **Port conflict** - 9443 already in use
   - Solution: Mapped to 9444 on host
4. **NPM proxy missing** - auth.schweitz.net not visible in UI
   - Solution: Entry was marked as deleted (is_deleted=1), recreated via UI

**NPM Configuration:**
- Created proxy host for auth.schweitz.net
- Forward to: http://localhost:9000
- SSL: Let's Encrypt (enforced, HSTS enabled)
- **Critical:** NO forward auth on auth.schweitz.net (prevents redirect loops)

### ✅ Milestone 2: Google OAuth Integration (COMPLETE)

**Google Cloud Console Setup:**
- Created OAuth credentials:
  - Client ID: `59195574918-813nsfslhjduqto8nc4a3ejg2lj133il.apps.googleusercontent.com`
  - Client Secret: `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`
  - Authorized redirect URI: `https://auth.schweitz.net/source/oauth/callback/google/`

**Authentik Configuration (via API):**
```python
# Created Google OAuth source
Source: "Google"
Slug: "google"
Provider: "google"
Consumer Key: [Google Client ID]
Consumer Secret: [Google Client Secret]
Enrollment Flow: default-source-enrollment
Authentication Flow: default-source-authentication
```

**Login Flow Configuration:**
- Updated `default-authentication-identification` stage
- Enabled "Show sources' labels"
- Added Google source to sources list
- Result: Google login button now appears on login page

**Testing Results:**
- ✅ Google login button visible on auth.schweitz.net
- ✅ OAuth redirect to Google works
- ✅ User created successfully: `jpmschweitzer@gmail.com`
- ✅ User type: `external` (correct for OAuth users)
- ⚠️  External users blocked from admin interface (expected behavior)
- ✅ Admin access via `akadmin` recovery key

**Enrollment Flow Issue & Resolution:**
- Initial error: "Flow does not apply to current user"
- Root cause: Browser session had conflicting flow plan cached
- Solution: Cleared cookies, used incognito window
- Policy check: `default-source-enrollment-if-sso` working correctly

### 🚧 Milestone 3: Forward Auth for Organizr (BLOCKED)

**Progress:**
- ✅ Created Proxy Provider "Organizr Proxy" via API
  - Mode: `forward_single`
  - External host: `https://home.schweitz.net`
  - Authorization flow: `default-provider-authorization-implicit-consent`
- ✅ Created Application "Organizr" via API
  - Slug: `organizr`
  - Provider: Organizr Proxy
  - Launch URL: `https://home.schweitz.net`
- ✅ Assigned provider to embedded outpost
- ✅ Embedded outpost responding on port 9444
  - Ping endpoint works: `https://localhost:9444/outpost.goauthentik.io/ping`

**Current Blocker:**
```
Issue: Auth endpoint returns 404
Endpoint: https://localhost:9444/outpost.goauthentik.io/auth/nginx
Status: 404 Not Found
Expected: 200 OK or 401/302 for unauthenticated requests

NPM Error Logs:
auth request unexpected status: 404 while sending to client
```

**Analysis:**
- Outpost is running and healthy
- Ping endpoint responds correctly
- Auth endpoint not being exposed by outpost
- Possible causes:
  1. Provider mode issue (`forward_single` vs `forward_domain`)
  2. Outpost not loading provider configuration
  3. Auth endpoint path incorrect for Authentik 2024.8.4
  4. Embedded outpost initialization incomplete

**Forward Auth Config Attempted:**
```nginx
# NPM advanced config for home.schweitz.net
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;

location /outpost.goauthentik.io {
    proxy_pass https://localhost:9444/outpost.goauthentik.io;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    # ... (additional headers)
}

location @goauthentik_proxy_signin {
    internal;
    return 302 /outpost.goauthentik.io/start?rd=$request_uri;
}
```

**Config Reverted:**
- Restored original NPM config for home.schweitz.net
- Organizr accessible without SSO (for now)
- Backup saved: `/data/nginx/proxy_host/2.conf.backup`

---

## Technical Details

### API Usage

Successfully used Authentik's REST API for automation:

```bash
# Created temporary API token
Token: dbc4eda544fd141a015b1ad1ec42955a4f6666fd22456a88c6f6402afa3107d1
Duration: 1 hour
User: akadmin

# API Endpoints Used:
POST /api/v3/providers/proxy/          # Create provider
POST /api/v3/core/applications/        # Create application
PATCH /api/v3/outposts/instances/{id}/ # Assign provider to outpost
GET /api/v3/flows/instances/           # List flows
```

### Database Operations

```sql
-- Created authentik database and user
CREATE DATABASE authentik;
CREATE USER authentik_user WITH PASSWORD 'F//j0ktck7cX06Vfgh0YXceONOtlSsHvadqROICeDx8=';
GRANT ALL PRIVILEGES ON DATABASE authentik TO authentik_user;
GRANT ALL ON SCHEMA public TO authentik_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO authentik_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO authentik_user;

-- Verified user creation
SELECT id, username, email, is_active, type
FROM authentik_core_user
WHERE email = 'jpmschweitzer@gmail.com';
-- Result: id=5, type=external, is_active=t

-- Checked OAuth source
SELECT slug, name, enabled, provider_type
FROM authentik_core_source s
LEFT JOIN authentik_sources_oauth_oauthsource o
ON s.policybindingmodel_ptr_id = o.source_ptr_id;
-- Result: slug=google, enabled=t, provider_type=google
```

### Memory Optimization Success

**Previous Failed Deployment:**
- Memory usage: 3-5GB
- Separate PostgreSQL instance: ~1GB
- Separate Redis instance: ~100MB
- No resource limits

**Current Deployment:**
```bash
$ docker stats authentik-server authentik-worker --no-stream
NAME               CPU %     MEM USAGE / LIMIT   MEM %
authentik-server   0.52%     291.1MiB / 512MiB   56.85%
authentik-worker   2.87%     271.9MiB / 384MiB   70.80%
Total: ~563MB

Savings: 82-88% reduction
Strategy:
  - Shared PostgreSQL (no dedicated instance)
  - Shared Redis (no dedicated instance)
  - Resource limits enforced
  - Single worker with 2 threads
  - Disabled: avatars, error reporting, footer links
  - Log level: warning
```

### Files Modified

1. **[stacks/authentik.yml](../../stacks/authentik.yml)** - Created
   - Authentik server and worker configuration
   - Shared infrastructure connections
   - Resource limits and health checks
   - Port mappings: 9000, 9444

2. **NPM Database** - Modified
   - Created proxy host for auth.schweitz.net
   - Attempted forward auth config (reverted)

3. **PostgreSQL** - Modified
   - Created authentik database
   - Created authentik_user with grants

4. **[STATUS.md](../../STATUS.md)** - Updated
   - Version: 0.8.0-authentik-sso
   - Active work: Security & SSO Implementation
   - Added Milestone 1 & 2 accomplishments
   - Documented Milestone 3 blocker

---

## Known Issues

### 1. Embedded Outpost Auth Endpoint Not Working

**Symptom:**
```
curl -k https://localhost:9444/outpost.goauthentik.io/auth/nginx
HTTP/1.1 404 Not Found
```

**Impact:**
- Cannot configure forward authentication for applications
- NPM forward auth results in 500 errors
- Applications remain unprotected

**Possible Solutions:**
1. **Change provider mode:**
   ```python
   # Update via Authentik UI: Applications → Providers → Organizr Proxy
   mode: "forward_domain"  # instead of "forward_single"
   cookie_domain: "schweitz.net"
   ```

2. **Deploy standalone outpost:**
   ```yaml
   # Add to authentik.yml or separate stack
   authentik-proxy:
     image: ghcr.io/goauthentik/proxy:2024.8.4
     environment:
       AUTHENTIK_HOST: https://auth.schweitz.net
       AUTHENTIK_TOKEN: <outpost-token>
     ports:
       - "9443:9443"
   ```

3. **Wait for full initialization:**
   - Monitor logs: `docker logs -f authentik-server`
   - Check outpost status in Authentik UI: System → Outposts
   - Verify provider assignment

4. **Investigate version compatibility:**
   - Authentik 2024.8.4 embedded outpost behavior
   - Check if auth endpoint requires specific configuration
   - Review Authentik documentation for forward auth setup

### 2. NPM Configuration Persistence

**Issue:**
- Database updates don't trigger nginx config regeneration
- Manual nginx file editing required
- Changes lost on NPM restart/update

**Workaround:**
- Update via NPM UI instead of database direct modification
- Keep backup of custom nginx configs
- Document config in code/scripts for reproducibility

---

## Next Steps

### Immediate (Milestone 3 Completion)

1. **Investigate Outpost Configuration:**
   - Check Authentik UI: System → Outposts → authentik Embedded Outpost
   - Verify provider is assigned and status is healthy
   - Review outpost logs for errors

2. **Try Provider Mode Change:**
   - Update Organizr Proxy provider to `forward_domain` mode
   - Add `cookie_domain: schweitz.net`
   - Restart Authentik containers
   - Test auth endpoint again

3. **Alternative: Deploy Standalone Outpost:**
   - Create outpost stack configuration
   - Generate outpost token in Authentik UI
   - Deploy container and test auth endpoint

4. **Test Forward Auth:**
   - Once auth endpoint works, apply NPM config
   - Test redirect to Authentik login
   - Verify SSO session persistence
   - Check for redirect loops

### Future Milestones (from security-implementation-plan.md)

- **M4:** Protect Core API with OIDC
- **M5:** Protect remaining services (9 services)
  - Jellyfin, Nextcloud, Gitea, Portainer, NPM, Uptime Kuma, Open WebUI, Netdata, Headscale
- **M6:** Documentation and rollback procedures

---

## Lessons Learned

### What Went Well

1. **Shared Infrastructure Approach:**
   - Massive memory savings (80-90% reduction)
   - Easier management (single PostgreSQL/Redis)
   - Successful from day 1

2. **API-Driven Configuration:**
   - Faster than UI clicks
   - Reproducible and documentable
   - Can be scripted for future deployments

3. **Incremental Testing:**
   - Validated each component before moving forward
   - Caught issues early (health checks, database permissions)
   - Easy to rollback when issues encountered

4. **Documentation During Implementation:**
   - Captured decisions and solutions in real-time
   - Easier to resume work later
   - Helpful for troubleshooting

### What Could Be Improved

1. **Version Research:**
   - Should have checked Authentik 2024.8.4 embedded outpost capabilities first
   - Version 2024.10+ has redirect loop issues (documented in security plan)
   - Tradeoff: stability vs features

2. **NPM Configuration Method:**
   - Direct database edits don't trigger config regeneration
   - Should have used NPM UI from start
   - Need better automation for NPM config management

3. **Testing Approach:**
   - Should have tested outpost endpoints before configuring NPM
   - Could have saved time on troubleshooting
   - Need outpost validation checklist

4. **Initialization Timing:**
   - Didn't account for embedded outpost startup delay
   - Should wait for full health before testing endpoints
   - Need patience with complex distributed systems

---

## References

- [Security Implementation Plan](../plans/active/security-implementation-plan.md)
- [Shared Infrastructure Architecture](../architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md)
- [Authentik Documentation](https://goauthentik.io/docs/)
- [NPM Backup](../../backups/npm-database-m0-20251120-152926.sqlite)
- [Authentik Stack](../../stacks/authentik.yml)

---

**Session End Status:**
- ✅ Authentik deployed and accessible
- ✅ Google OAuth fully functional
- ⚠️  Forward auth blocked on outpost initialization
- 🔄 Investigation continuing in next session