diff --git a/docs/AUTHENTIK_FORWARD_AUTH_DEPLOYMENT.md b/docs/AUTHENTIK_FORWARD_AUTH_DEPLOYMENT.md deleted file mode 100644 index e880549..0000000 --- a/docs/AUTHENTIK_FORWARD_AUTH_DEPLOYMENT.md +++ /dev/null @@ -1,373 +0,0 @@ -# Authentik Forward Authentication Deployment Summary - -## Overview - -Successfully deployed Authentik forward authentication to all NPM-managed domains. All protected sites now require SSO authentication via Authentik before access is granted. - -## Deployment Date - -2025-11-16 - -## Architecture - -### Components - -1. **Authentik Server** (`authentik-server`) - - Port: 9000 - - Container: `authentik-server` - - Purpose: Main authentication server - -2. **Authentik Proxy Outpost** (`authentik-proxy-outpost`) - - Port: 9001 (HTTP), 9300 (Metrics) - - Container: `authentik-proxy-outpost` - - Purpose: Forward authentication endpoint for NPM - - Network: host mode (to match NPM) - - Image: `ghcr.io/goauthentik/proxy:latest` - -3. **Nginx Proxy Manager** (`nginx-proxy-manager`) - - Ports: 80, 81, 443 - - Purpose: Reverse proxy with SSL termination - - Forward auth endpoint: `http://192.168.86.149:9001/outpost.goauthentik.io/auth/nginx` - -### Authentication Flow - -``` -1. User → https://home.schweitz.net -2. NPM → auth_request to http://192.168.86.149:9001/outpost.goauthentik.io/auth/nginx -3. Outpost → validates session with Authentik Server (port 9000) -4. If not authenticated: - a. NPM → redirects to /outpost.goauthentik.io/start?rd=https://home.schweitz.net/ - b. Outpost → redirects to https://auth.schweitz.net/if/flow/... - c. User → logs in via Google OAuth - d. Authentik → sets session cookie - e. Redirect → back to https://home.schweitz.net/ -5. If authenticated: - a. Request passes through with user headers -``` - -## Protected Domains - -All NPM proxy hosts now require authentication: - -1. **192.168.86.149** - Direct IP access -2. **amp.schweitz.net** - AMP Server -3. **cloud.schweitz.net** - Nextcloud -4. **code.schweitz.net** - VS Code Server -5. **git.schweitz.net** - Gitea -6. **home.schweitz.net** - Home Assistant/Dashboard -7. **media.schweitz.net** - Media Server -8. **tatlock.schweitz.net** - Tatlock Services -9. **tower-of-joy** - Tower of Joy Services - -### Excluded Domains - -- **auth.schweitz.net** - Authentik itself (cannot protect the auth provider) - -## Configuration Details - -### Authentik Proxy Provider - -- **Name**: `npm-forward-auth-provider` -- **ID**: 2 -- **Mode**: `forward_single` -- **External Host**: `https://auth.schweitz.net` -- **Token Validity**: 480 minutes (8 hours) -- **Session Duration**: 480 minutes (8 hours) -- **Rolling Expiration**: Active (sessions extend on use) - -### Authentik Outpost - -- **Name**: `npm-forward-auth-outpost` -- **ID**: `5fbba1f4-e50f-4d3f-94e8-8d9f30bf7a51` -- **Type**: `proxy` -- **Provider**: `npm-forward-auth-provider` (ID: 2) -- **Token**: `aVJXnn3I44NhTYRYxGtbP3kJLEKQrTEMwu9IPVrhzN3NZV4vakgztcEf2L44` -- **Container**: `authentik-proxy-outpost` -- **Listen**: `0.0.0.0:9001` (HTTP), `0.0.0.0:9300` (metrics) - -### NPM Configuration - -Each protected proxy host has this advanced nginx configuration: - -```nginx -# Authentik Forward Authentication -# Send authentication requests to Authentik -auth_request /outpost.goauthentik.io/auth/nginx; - -# Preserve authentication cookies -auth_request_set $auth_cookie $upstream_http_set_cookie; -add_header Set-Cookie $auth_cookie; - -# Get user information from Authentik -auth_request_set $authentik_username $upstream_http_x_authentik_username; -auth_request_set $authentik_groups $upstream_http_x_authentik_groups; -auth_request_set $authentik_email $upstream_http_x_authentik_email; -auth_request_set $authentik_name $upstream_http_x_authentik_name; -auth_request_set $authentik_uid $upstream_http_x_authentik_uid; - -# Pass user info to backend -proxy_set_header X-authentik-username $authentik_username; -proxy_set_header X-authentik-groups $authentik_groups; -proxy_set_header X-authentik-email $authentik_email; -proxy_set_header X-authentik-name $authentik_name; -proxy_set_header X-authentik-uid $authentik_uid; - -# On authentication failure, redirect to Authentik login -error_page 401 = @authentik_proxy_signin; - -location @authentik_proxy_signin { - internal; - add_header Set-Cookie $auth_cookie; - return 302 /outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri; -} - -# Authentik authentication endpoint -location /outpost.goauthentik.io { - proxy_pass http://192.168.86.149:9001/outpost.goauthentik.io; - proxy_set_header X-Original-URL $scheme://$http_host$request_uri; - proxy_pass_request_body off; - proxy_set_header Content-Length ""; - proxy_set_header Host $host; -} -``` - -## Deployment Steps Taken - -### 1. Created Proxy Provider - -```bash -docker exec core-api /venv/bin/python /app/setup_authentik_forward_auth.py -``` - -- Created `npm-forward-auth-provider` proxy provider (ID: 2) -- Created `NPM Forward Auth` application (slug: `npm-forward-auth`) -- Updated embedded outpost to include provider - -### 2. Enabled Forward Auth on NPM Hosts - -```bash -docker exec core-api /venv/bin/python /app/enable_npm_forward_auth.py -``` - -- Added forward auth configuration to all 9 NPM proxy hosts -- Initially pointed to `http://192.168.86.149:9000` (incorrect) - -### 3. Created Proxy Outpost - -```bash -python3 -c "..." # Created outpost via Authentik API -``` - -- Created `npm-forward-auth-outpost` (ID: `5fbba1f4-e50f-4d3f-94e8-8d9f30bf7a51`) -- Authentik auto-created container via Docker service connection -- Retrieved token from auto-created container environment variables - -### 4. Deployed Proxy Outpost Container - -```bash -docker run -d \ - --name authentik-proxy-outpost \ - --restart unless-stopped \ - --network host \ - -e AUTHENTIK_HOST=http://192.168.86.149:9000 \ - -e AUTHENTIK_TOKEN=aVJXnn3I44NhTYRYxGtbP3kJLEKQrTEMwu9IPVrhzN3NZV4vakgztcEf2L44 \ - -e AUTHENTIK_LISTEN__HTTP=0.0.0.0:9001 \ - -e AUTHENTIK_LISTEN__METRICS=0.0.0.0:9300 \ - ghcr.io/goauthentik/proxy:latest -``` - -### 5. Updated NPM Hosts to Port 9001 - -```bash -docker exec core-api /venv/bin/python /app/fix_port_to_9001.py -``` - -- Updated all 9 hosts to point to `http://192.168.86.149:9001` instead of port 9000 - -## Verification - -### Testing Forward Auth - -```bash -# Test ping endpoint -curl http://192.168.86.149:9001/outpost.goauthentik.io/ping -# Expected: 204 No Content - -# Test unauthenticated request -curl -I http://home.schweitz.net -# Expected: 302 redirect to /outpost.goauthentik.io/start?rd=... -``` - -### Logs - -```bash -# Outpost logs -docker logs authentik-proxy-outpost - -# Authentik server logs -docker logs authentik-server - -# NPM logs -docker logs nginx-proxy-manager -``` - -## Token & Session Management - -### Token Validity - -- **Initial validity**: 480 minutes (8 hours) -- **Rolling expiration**: Token stays active on use -- **Idle timeout**: 8 hours after last activity -- **Explicit logout**: Token invalidated immediately at https://auth.schweitz.net/if/user/#/settings - -### Session Behavior - -- **Single login** protects all domains under `*.schweitz.net` -- **Cookie domain**: Shared across all sites (set by individual domain) -- **Persistent**: Survives browser restarts until expiration/logout -- **Secure**: HTTPS-only, HttpOnly flag set - -## User Management - -### Adding Users - -1. Go to https://auth.schweitz.net/if/admin/ -2. Navigate to **Directory** → **Users** -3. Click **Create** → **Create and enroll user** -4. Enter user details -5. Send enrollment invite - -### Current Access Control - -**All authenticated users** can access all protected sites. To restrict: - -1. Go to https://auth.schweitz.net/if/admin/#/core/applications -2. Select **NPM Forward Auth** application -3. Go to **Policy / Group / User Bindings** -4. Add specific users or groups - -## Troubleshooting - -### Check Outpost Status - -```bash -docker ps | grep authentik-proxy-outpost -docker logs authentik-proxy-outpost --tail 50 -``` - -### Check Authentik Server - -```bash -docker logs authentik-server --tail 50 | grep -i outpost -``` - -### Test Endpoints - -```bash -# Ping (should return 204) -curl -I http://192.168.86.149:9001/outpost.goauthentik.io/ping - -# Auth endpoint (needs proper headers from nginx) -curl -I http://home.schweitz.net -``` - -### Common Issues - -1. **502 Bad Gateway**: Outpost not running or not reachable - - Check: `docker ps | grep authentik-proxy-outpost` - - Fix: Restart outpost container - -2. **Redirect Loop**: Misconfigured nginx or outpost - - Check: NPM advanced config points to port 9001 - - Check: Outpost logs for errors - -3. **Session Expires Too Quickly**: Token validity too short - - Check: Provider token_validity setting (should be 480 minutes) - -## Automation Scripts - -### Enable Forward Auth on New Sites - -```python -from src.clients.npm_client import get_npm_client - -npm = get_npm_client() -proxy = await npm.create_proxy_host( - domain_names=["newsite.schweitz.net"], - forward_host="backend-container", - forward_port=8080, - ssl_enabled=True -) -await npm.enable_authentik_forward_auth(proxy["id"]) -``` - -### Bulk Operations - -```bash -# Enable on all hosts -docker exec core-api /venv/bin/python /app/enable_npm_forward_auth.py - -# Update to port 9001 -docker exec core-api /venv/bin/python /app/fix_port_to_9001.py -``` - -## Security Considerations - -1. ✓ **HTTPS Required**: All sites use SSL termination at NPM -2. ✓ **Secure Cookies**: HttpOnly and Secure flags prevent XSS/MITM -3. ✓ **Token Rotation**: Tokens extend on activity for security -4. ✓ **Audit Logging**: Authentik logs all authentication events -5. ⚠ **MFA**: Not yet enabled (future enhancement) - -## Monitoring - -### Health Checks - -```bash -# Authentik server -curl http://192.168.86.149:9000/-/health/live/ - -# Proxy outpost -curl http://192.168.86.149:9001/outpost.goauthentik.io/ping - -# Metrics -curl http://192.168.86.149:9300/metrics -``` - -### Authentik Admin Dashboard - -- **URL**: https://auth.schweitz.net/if/admin/ -- **Events**: System → Events (view login attempts, failures) -- **Active Sessions**: System → Tokens -- **Outpost Status**: System → Outposts - -## Future Enhancements - -1. **Multi-Factor Authentication**: Enable TOTP/SMS for high-security sites -2. **Per-Site Access Control**: Different user groups for different domains -3. **Rate Limiting**: Prevent brute force attacks -4. **IP Whitelisting**: Allow certain IPs without auth -5. **API Key Support**: Service-to-service authentication -6. **Integration with Core API**: Expose auth status via REST API - -## Related Documentation - -- **Main Documentation**: `FORWARD_AUTH_CONFIGURATION.md` -- **SSO Progress**: `SSO_IMPLEMENTATION_PROGRESS.md` -- **Security Plan**: `security-implementation-plan.md` -- **Authentik Docs**: https://docs.goauthentik.io/docs/providers/proxy/ - -## Notes - -- The embedded Authentik outpost (ID: `fbbefe20-b2e1-4706-8533-7bfece51989c`) is for Authentik's own web UI, not for external forward auth -- Forward auth requires a separate proxy outpost container listening on its own port (9001) -- The outpost automatically created by Authentik via Docker service connection had the wrong port configuration -- We manually deployed the outpost with correct settings using `docker run` -- NPM uses host network mode, so the outpost must also use host mode or be accessible via host IP:port -- Token keys are only visible once during creation in the Authentik UI; we retrieved it from the auto-created container's environment variables - -## Completion Status - -✅ **COMPLETE** - All protected sites now require Authentik authentication - -**Deployment successful as of 2025-11-16 15:41 UTC** diff --git a/docs/FORWARD_AUTH_CONFIGURATION.md b/docs/FORWARD_AUTH_CONFIGURATION.md deleted file mode 100644 index 5919e0c..0000000 --- a/docs/FORWARD_AUTH_CONFIGURATION.md +++ /dev/null @@ -1,259 +0,0 @@ -# Authentik Forward Authentication Configuration - -## Overview - -All NPM-managed domains now use Authentik for centralized Single Sign-On (SSO) authentication. When users access any protected site, they are automatically redirected to Authentik for login, then returned to their original destination. - -## Authentication Flow - -1. **User accesses protected site** (e.g., https://portainer.schweitz.net) -2. **NPM checks authentication** via Authentik forward auth -3. **If not authenticated**: Redirect to https://auth.schweitz.net/outpost.goauthentik.io/start -4. **User logs in** with Google (via Authentik) -5. **Authentik sets session cookie** (8-hour validity) -6. **Redirect back** to originally requested URL -7. **Subsequent requests**: Automatically authenticated (no login required) - -## Token & Session Configuration - -### Token Validity -- **Initial validity**: 8 hours (480 minutes) -- **Activity-based extension**: Token stays active on use (rolling expiration) -- **Idle timeout**: 8 hours after last activity -- **Explicit logout**: Token invalidated immediately - -### Session Behavior -- **Single login** protects all domains under `*.schweitz.net` -- **Cookie domain**: Shared across all sites -- **Persistent**: Survives browser restarts (until expiration/logout) -- **Secure**: HTTPS-only, HttpOnly flag set - -## Protected Domains - -The following domains are protected with Authentik forward auth: - -1. **192.168.86.149** - Direct IP access -2. **amp.schweitz.net** - AMP Server -3. **cloud.schweitz.net** - Nextcloud -4. **code.schweitz.net** - VS Code Server -5. **git.schweitz.net** - Gitea -6. **home.schweitz.net** - Home Assistant/Dashboard -7. **media.schweitz.net** - Media Server -8. **tatlock.schweitz.net** - Tatlock Services -9. **tower-of-joy** - Tower of Joy Services - -### Unprotected Domains - -- **auth.schweitz.net** - Authentik itself (cannot protect the auth provider) - -## User Management - -### Adding Users - -1. Go to https://auth.schweitz.net/if/admin/ -2. Navigate to **Directory** → **Users** -3. Click **Create** → **Create and enroll user** -4. Enter user details -5. Send enrollment invite (they'll set up Google OAuth) - -### User Access Control - -Currently, **all authenticated users** can access protected sites. To restrict access: - -1. Go to https://auth.schweitz.net/if/admin/#/core/applications -2. Select **NPM Forward Auth** application -3. Go to **Policy / Group / User Bindings** -4. Add specific users or groups - -### Group-Based Access (Future) - -You can create groups and assign different access levels: -- `admin` - Full access to all sites -- `family` - Access to media, home -- `developers` - Access to code, git - -## Logout - -Users can log out at: https://auth.schweitz.net/if/user/#/settings - -Click **Sign Out** to invalidate the session across all protected sites. - -## Technical Implementation - -### Authentik Components - -1. **Proxy Provider** (`npm-forward-auth-provider`) - - Mode: `forward_single` - - External Host: `https://auth.schweitz.net` - - Token Validity: 480 minutes - -2. **Application** (`NPM Forward Auth`) - - Links provider to user interface - - Accessible at: https://auth.schweitz.net - -3. **Outpost** (`authentik Embedded Outpost`) - - Handles authentication requests from NPM - - Endpoint: `http://authentik-server:9000/outpost.goauthentik.io` - -### NPM Configuration - -Each proxy host has advanced nginx configuration: - -```nginx -# Forward auth to Authentik -auth_request /outpost.goauthentik.io/auth/nginx; - -# Preserve cookies -auth_request_set $auth_cookie $upstream_http_set_cookie; -add_header Set-Cookie $auth_cookie; - -# Extract user info -auth_request_set $authentik_username $upstream_http_x_authentik_username; -auth_request_set $authentik_groups $upstream_http_x_authentik_groups; -auth_request_set $authentik_email $upstream_http_x_authentik_email; - -# Pass to backend -proxy_set_header X-authentik-username $authentik_username; -proxy_set_header X-authentik-email $authentik_email; - -# Redirect on auth failure -error_page 401 = @authentik_proxy_signin; - -location @authentik_proxy_signin { - return 302 /outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri; -} - -# Auth endpoint -location /outpost.goauthentik.io { - proxy_pass http://authentik-server:9000/outpost.goauthentik.io; - proxy_set_header X-Original-URL $scheme://$http_host$request_uri; - proxy_pass_request_body off; -} -``` - -## Backend Application Integration - -Protected applications receive user information via headers: - -- `X-authentik-username` - Username -- `X-authentik-email` - Email address -- `X-authentik-groups` - Comma-separated groups -- `X-authentik-name` - Full name -- `X-authentik-uid` - Unique user ID - -Applications can use these headers for: -- Displaying user info -- Audit logging -- Role-based access control -- Personalization - -## Troubleshooting - -### User Can't Log In - -1. **Check Authentik status**: `docker ps | grep authentik` -2. **Check Authentik logs**: `docker logs authentik-server` -3. **Verify Google OAuth**: - - Go to https://auth.schweitz.net/if/admin/#/core/sources - - Ensure Google source is enabled -4. **Check user exists**: - - Go to https://auth.schweitz.net/if/admin/#/identity/users - - Verify user account is active - -### Redirect Loop - -If users get stuck in a redirect loop: - -1. **Clear browser cookies** for `*.schweitz.net` -2. **Check NPM config**: Ensure `/outpost.goauthentik.io` location exists -3. **Restart NPM**: `docker restart npm` -4. **Check outpost**: Verify provider is assigned to outpost - -### 502 Bad Gateway - -If auth requests fail: - -1. **Check Authentik container**: `docker ps | grep authentik` -2. **Verify network**: Ensure NPM can reach `authentik-server:9000` -3. **Check NPM logs**: `docker logs npm` - -### Session Expires Too Quickly - -If users are logged out unexpectedly: - -1. **Check token validity**: Should be 480 minutes (8 hours) -2. **Verify rolling expiration**: Active use should extend session -3. **Check system time**: Ensure Docker host time is correct - -## Automation - -### Adding Forward Auth to New Sites - -When creating new proxy hosts via core-api: - -```python -from src.clients.npm_client import get_npm_client - -npm = get_npm_client() - -# Create proxy host -proxy = await npm.create_proxy_host( - domain_names=["newsite.schweitz.net"], - forward_host="backend-container", - forward_port=8080, - ssl_enabled=True -) - -# Enable forward auth -await npm.enable_authentik_forward_auth(proxy["id"]) -``` - -### Bulk Enable/Disable - -To enable on all hosts: -```bash -docker exec core-api /venv/bin/python /app/enable_npm_forward_auth.py -``` - -## Security Considerations - -1. **HTTPS Required**: Forward auth should only be used with HTTPS -2. **Secure Cookies**: HttpOnly and Secure flags prevent XSS/MITM -3. **Token Rotation**: Tokens are rotated on activity for security -4. **Audit Logging**: Authentik logs all authentication events -5. **MFA Support**: Can be enabled in Authentik for additional security - -## Monitoring - -### Check Authentication Status - -```bash -# Check Authentik health -curl http://authentik-server:9000/-/health/live/ - -# Check active sessions (in Authentik admin) -# https://auth.schweitz.net/if/admin/#/events/log -``` - -### Monitor Failed Attempts - -Go to **System** → **Events** in Authentik admin to see: -- Failed login attempts -- Successful authentications -- Token expirations -- Policy violations - -## Future Enhancements - -1. **Per-Site Access Control**: Different user groups for different domains -2. **Multi-Factor Authentication**: SMS/TOTP for high-security sites -3. **Rate Limiting**: Prevent brute force attacks -4. **IP Whitelisting**: Allow certain IPs without auth -5. **API Key Support**: Service-to-service authentication - -## Related Documentation - -- **Security Implementation Plan**: `security-implementation-plan.md` -- **SSO Progress**: `SSO_IMPLEMENTATION_PROGRESS.md` -- **OIDC Configuration**: `OIDC_CONFIGURATION.md` -- **Authentik Docs**: https://docs.goauthentik.io/docs/providers/proxy/ diff --git a/docs/OIDC_CONFIGURATION.md b/docs/OIDC_CONFIGURATION.md deleted file mode 100644 index bdad7a7..0000000 --- a/docs/OIDC_CONFIGURATION.md +++ /dev/null @@ -1,167 +0,0 @@ -# OIDC Authentication Configuration - -## Overview - -Core-API has been configured with OIDC authentication support using Authentik as the identity provider. This provides secure authentication for infrastructure management endpoints. - -## Current Status - -**OIDC is currently DISABLED** (`oidc_enabled=false` in config.py) - -This allows: -- Internal services to access the API without authentication -- Direct API access from the Docker network -- Backward compatibility with existing integrations - -## Authentik Configuration - -### Provider Details - -- **Provider Name**: `core-api-provider` -- **Provider ID**: `1` -- **Client ID**: `core-api` -- **Client Secret**: `WfsY0iIVOmO1wvXn1u8jwa08eiQsn2yVf9toBwVyEqps3M98nOZwJMgDOqH5PNZGM6wIxKTwlwYemtaOUg9u5bocd0EqShyoe4yhBQq4SDd0svyArILHZeGHFVEqUi4d` -- **Issuer**: `https://auth.schweitz.net/application/o/core-api/` - -### Application Details - -- **Application Name**: `Core API` -- **Slug**: `core-api` -- **Launch URL**: `http://localhost:8083/docs` - -### Redirect URIs - -- `http://localhost:8083/docs/oauth2-redirect` -- `https://core-api.schweitz.net/docs/oauth2-redirect` -- `http://192.168.86.149:8083/docs/oauth2-redirect` - -## Enabling OIDC Authentication - -When ready to enable OIDC authentication for external access: - -### 1. Update core-api Configuration - -In `/home/jpmschweitzer/Projects/portainer-core/services/core-api/src/config.py`: - -```python -# OIDC Authentication (Authentik) -oidc_enabled: bool = True # Change from False to True -oidc_issuer: str = "https://auth.schweitz.net/application/o/core-api/" -oidc_audience: str = "core-api" -oidc_client_secret: str = "WfsY0iIVOmO1wvXn1u8jwa08eiQsn2yVf9toBwVyEqps3M98nOZwJMgDOqH5PNZGM6wIxKTwlwYemtaOUg9u5bocd0EqShyoe4yhBQq4SDd0svyArILHZeGHFVEqUi4d" -``` - -### 2. Restart core-api - -```bash -docker restart core-api -``` - -### 3. Test Authentication - -1. Visit: http://localhost:8083/docs -2. Click the "Authorize" button -3. Log in with Google via Authentik -4. Access protected endpoints - -## Protected Endpoints - -When OIDC is enabled, the following endpoints require admin authentication: - -### Infrastructure Management (Write Operations) -- `POST /infrastructure/services` - Deploy stack -- `PUT /infrastructure/services/{name}` - Update stack -- `DELETE /infrastructure/services/{name}` - Delete stack -- `POST /infrastructure/proxy` - Create proxy host -- `POST /infrastructure/services/{name}/stop` - Stop service -- `POST /infrastructure/services/{name}/start` - Start service - -### Monitoring Management -- `POST /infrastructure/monitors` - Create monitor -- `PUT /infrastructure/monitors/{monitor_id}` - Update monitor -- `DELETE /infrastructure/monitors/{monitor_id}` - Delete monitor - -### Read Endpoints (Public) -All GET endpoints remain publicly accessible: -- `/health` - Health check -- `/infrastructure/health` - Infrastructure health -- `/infrastructure/services` - List services -- `/infrastructure/ports` - List ports -- `/infrastructure/domains` - List domains -- `/infrastructure/monitors` - List monitors - -## Internal Network Access - -**Important**: After enabling OIDC for external users, internal services still need unrestricted access. - -### Future Enhancement Options - -1. **Network-based Authentication** - - Check if request originates from `docker-dataplane` network - - Allow requests from internal IPs without auth - - Require OIDC only for external requests - -2. **Service Accounts** - - Create machine tokens for internal services - - Use Bearer token authentication for service-to-service calls - - Keep OIDC for user authentication - -3. **NPM Proxy Layer** - - External domain (https://core-api.schweitz.net) → Requires Authentik SSO - - Internal access (http://core-api:8083) → No authentication - -## Admin Groups - -Users must be members of one of these Authentik groups to access protected endpoints: -- `admin` -- `authentik Admins` - -Configure user group membership in Authentik admin panel: -https://auth.schweitz.net/if/admin/#/identity/users - -## Authentik Management - -- **Admin Panel**: https://auth.schweitz.net/if/admin/ -- **Applications**: https://auth.schweitz.net/if/admin/#/core/applications -- **Providers**: https://auth.schweitz.net/if/admin/#/core/providers - -## API Token Management - -The Authentik API token used for automation is stored in: -`/home/jpmschweitzer/Projects/portainer-core/services/core-api/src/credentials.py` - -```python -AUTHENTIK_CORE_API_TOKEN = "peXM0EzDv2Wiwycbfm3cE5O44IMAOR8ntUwKVP977yFvVopCzDlKY8tymlMM" -``` - -To create additional tokens: https://auth.schweitz.net/if/admin/#/identity/tokens - -## Security Considerations - -1. **Client Secret**: Stored in config.py (gitignored), consider moving to environment variable -2. **Token Validation**: Tokens are validated using JWKS from Authentik -3. **Token Expiry**: Access tokens valid for 60 minutes, refresh tokens for 30 days -4. **SSL**: All production endpoints should use HTTPS (via NPM) -5. **Admin Access**: Restrict admin group membership carefully - -## Troubleshooting - -### Login Issues -- Check Authentik service status: `docker ps | grep authentik` -- Check Authentik logs: `docker logs authentik-server` -- Verify redirect URIs match exactly in Authentik provider config - -### Token Validation Errors -- Verify OIDC issuer URL is correct -- Check core-api logs: `docker logs core-api` -- Ensure Authentik is accessible from core-api container - -### Permission Denied (403) -- Verify user is member of `admin` group in Authentik -- Check user claims in JWT token at https://jwt.io - -## Related Documentation - -- **Security Implementation Plan**: `/home/jpmschweitzer/Projects/portainer-core/security-implementation-plan.md` -- **SSO Progress**: `/home/jpmschweitzer/Projects/portainer-core/SSO_IMPLEMENTATION_PROGRESS.md` -- **Authentik Docs**: https://docs.goauthentik.io/ diff --git a/docs/SSO_IMPLEMENTATION_PROGRESS.md b/docs/SSO_IMPLEMENTATION_PROGRESS.md deleted file mode 100644 index 207d22d..0000000 --- a/docs/SSO_IMPLEMENTATION_PROGRESS.md +++ /dev/null @@ -1,115 +0,0 @@ -# SSO Implementation Progress Tracker - -**Project:** Google OAuth SSO for Homelab Infrastructure -**Started:** 2025-11-15 -**Status:** 🚧 Phase 1 - In Progress - ---- - -## Overview - -Implementing Single Sign-On (SSO) using: -- **Identity Provider:** Authentik -- **Authentication Source:** Google OAuth (Workspace + Gmail) -- **In-Scope Services:** core-api, Nextcloud, Jellyfin, Gitea, Open WebUI, Organizr, code-server - ---- - -## Phase 1: Foundation (Week 1) -**Goal:** Deploy Authentik, configure Google OAuth, protect core-api - -### Task Checklist - -- [x] **1.1 Deploy Authentik Stack** *(In Progress)* - - [x] Create `stacks/authentik.yml` - - [x] Generate secrets (PostgreSQL password, Authentik secret key) - - [x] Create `.env.authentik` file - - [x] Create data directories - - [ ] Deploy via Portainer - - [ ] Verify services running (4/4: server, worker, postgresql, redis) - - [ ] Complete initial setup wizard - - [ ] Access admin portal - -- [ ] **1.2 Configure NPM Proxy** - - [ ] Create proxy host: `auth.schweitz.net` → `authentik-server:9000` - - [ ] Enable SSL with Let's Encrypt - - [ ] Test HTTPS access - - [ ] Verify health endpoint - -- [ ] **1.3 Google OAuth Setup** - - [ ] Create/configure Google Cloud Project - - [ ] Set up OAuth consent screen - - [ ] Create OAuth 2.0 credentials - - [ ] Note Client ID and Client Secret - - [ ] Configure authorized redirect URIs - -- [ ] **1.4 Configure Google Source in Authentik** - - [ ] Add Google OAuth source - - [ ] Configure scopes: openid, email, profile - - [ ] Test login with Google Workspace account - - [ ] Test login with Gmail account - - [ ] Verify user profile synced - -- [ ] **1.5 Implement core-api OIDC Authentication** - - [ ] Add dependencies: PyJWT, python-jose - - [ ] Create `src/auth/oidc.py` module - - [ ] Update `src/config.py` with OIDC settings - - [ ] Create Authentik OIDC provider for core-api - - [ ] Protect infrastructure endpoints - - [ ] Update OpenAPI docs with security scheme - -- [ ] **1.6 Testing & Validation** - - [ ] Test unauthenticated API request (expect 401) - - [ ] Test authenticated API request with valid token - - [ ] Verify user claims available in endpoints - - [ ] Test token expiration handling - - [ ] Test admin-only endpoints - - [ ] Update widget for OAuth flow - ---- - -## Progress Log - -### 2025-11-15 - 21:20 CET - -**[Completed]** Shared Infrastructure Architecture -- ✅ Designed shared PostgreSQL + Redis architecture -- ✅ Created `SHARED_INFRASTRUCTURE_ARCHITECTURE.md` documentation -- ✅ Created separate stacks for modularity: - - `postgres-shared.yml` (centralized database) - - `redis-shared.yml` (centralized cache) - - `authentik-shared.yml` (using shared backends) -- ✅ Generated secure credentials (all passwords 32-byte random) -- ✅ Created unified `docker-dataplane` network -- ✅ Created PostgreSQL init script for multi-database setup -- ✅ Created environment files: - - `.env.postgres` (PostgreSQL + app DB passwords) - - `.env.authentik-shared` (Authentik config) - -**Architecture Benefits:** -- Resource savings: ~400MB RAM per service using shared infrastructure -- Centralized backups and monitoring -- Easier maintenance and upgrades -- Modular deployment (PostgreSQL and Redis as separate stacks) - -**[Next]** Deploy shared infrastructure, then Authentik - ---- - -## Next: Create Authentik Stack - -Creating `stacks/authentik.yml` with: -- authentik-server -- authentik-worker -- PostgreSQL database -- Redis cache - -Expected resources: ~500MB RAM, 1.5 CPU, 5GB storage - ---- - -## Notes - -- **Architecture Decision:** FastAPI native OIDC for core-api (not NPM forward auth) -- **Out of Scope:** Infrastructure tools, data providers, local services -- **Security:** All passwords/secrets via environment variables, not committed to git diff --git a/security-implementation-plan.md b/docs/security-implementation-plan.md similarity index 100% rename from security-implementation-plan.md rename to docs/security-implementation-plan.md diff --git a/organizr-widgets/README.md b/organizr-widgets/README.md index ba1cb4b..9658ad4 100644 --- a/organizr-widgets/README.md +++ b/organizr-widgets/README.md @@ -162,7 +162,7 @@ cors_origins: list[str] = ["http://your-organizr-domain.com"] The following service groups are defined (stopping one stops all in group): - **jellyfin**: jellyfin -- **nextcloud**: nextcloud, nextcloud-db, nextcloud-redis +- **nextcloud**: nextcloud (uses shared postgres-shared + redis-shared) - **gitea**: gitea, gitea-db - **ai-stack**: open-webui, ollama, qdrant - **samba**: samba @@ -179,6 +179,8 @@ These infrastructure services are protected: - watchtower - netdata - maintenance +- postgres-shared (shared database infrastructure) +- redis-shared (shared cache infrastructure) ## Advanced: Customizing the UI diff --git a/services/core-api/REFACTORING_PLAN.md b/services/core-api/REFACTORING_PLAN.md deleted file mode 100644 index e3e461d..0000000 --- a/services/core-api/REFACTORING_PLAN.md +++ /dev/null @@ -1,332 +0,0 @@ -# Core-API Refactoring Plan - -**Date:** 2025-11-14 -**Goal:** Restructure Core-API into controller-based architecture and add Infrastructure Management API - -## Current Structure - -``` -src/ -├── api/ -│ └── v1/ -│ ├── chat.py # AI chat completions -│ ├── models.py # Model listing -│ ├── conversations.py # Conversation memory -│ └── schemas.py # Pydantic schemas -├── web_scraper/ -│ ├── router.py # Webscraper endpoints -│ ├── service.py -│ └── schemas.py -├── models/ -│ ├── ollama_client.py # Ollama HTTP client -│ └── embeddings.py -├── memory/ # Memory tier system -├── config.py # Global settings -└── main.py # FastAPI app - -``` - -## Target Structure - -``` -src/ -├── controllers/ # NEW: Controller-based routing -│ ├── __init__.py -│ ├── base.py # Base controller class -│ ├── ai_controller.py # AI Orchestrator (chat, models, conversations) -│ ├── tools_controller.py # Utility tools (webscraper, etc.) -│ ├── health_controller.py # Health & monitoring -│ └── infrastructure_controller.py # Infrastructure automation -├── clients/ # NEW: External API clients -│ ├── __init__.py -│ ├── portainer_client.py # Portainer API -│ ├── npm_client.py # Nginx Proxy Manager API -│ └── kuma_client.py # Uptime Kuma Socket.IO API -├── api/v1/ # Keep existing for backward compat -├── web_scraper/ # Keep as-is for now -├── models/ # Keep as-is -├── memory/ # Keep as-is -├── config.py # Enhanced with infrastructure settings -└── main.py # Updated routing - -``` - -## Implementation Phases - -### Phase 1: Infrastructure Setup ✅ COMPLETE -- [x] Research API authentication methods -- [x] Add infrastructure settings to config.py -- [x] Create credentials.py for sensitive data (gitignored) -- [x] Create credentials.example.py as template -- [x] Update .gitignore to exclude credentials.py -- [x] Update config.py to import from credentials module -- [x] Create /controllers directory structure -- [x] Create /clients directory structure -- [x] Create base controller class - -### Phase 2: API Clients ✅ COMPLETE (Portainer & NPM) -- [x] Implement Portainer API client (access token auth) -- [x] Implement NPM API client (JWT with refresh) -- [x] Add token storage/refresh mechanisms -- [ ] Implement Uptime Kuma Socket.IO client (DEFERRED - WebSocket complexity) - -### Phase 3: Infrastructure Controller ✅ COMPLETE -- [x] GET /infrastructure/health - Check connectivity ✅ TESTED -- [x] GET /infrastructure/services - List all services ✅ TESTED -- [x] GET /infrastructure/services/{name} - Get service details ✅ TESTED -- [x] GET /infrastructure/ports - List allocated ports ✅ IMPLEMENTED & TESTED -- [x] GET /infrastructure/domains - List configured domains ✅ TESTED -- [x] Integrate with main.py routing ✅ TESTED -- [x] Fix Pydantic validation issues (status field type conversion) -- [x] POST /infrastructure/services - Deploy new service ✅ TESTED -- [x] PUT /infrastructure/services/{name} - Update service ✅ TESTED -- [x] DELETE /infrastructure/services/{name} - Remove service ✅ TESTED -- [x] POST /infrastructure/proxy - Create NPM proxy host with optional SSL ✅ IMPLEMENTED -- [ ] POST /infrastructure/monitoring/add - Auto-add Kuma monitor (DEFERRED - Socket.IO complexity) - -### Phase 4: Refactor Existing Controllers ✅ COMPLETE -- [x] Move AI endpoints to ai_controller.py ✅ COMPLETE -- [x] Move webscraper to tools_controller.py ✅ COMPLETE -- [x] Move health check to health_controller.py ✅ COMPLETE -- [x] Update main.py imports and routing ✅ COMPLETE -- [x] Test all refactored endpoints ✅ ALL WORKING - -### Phase 5: Testing & Documentation ✅ COMPLETE -- [x] Test all refactored endpoints ✅ ALL WORKING -- [x] Update API documentation (OpenAPI spec auto-generated and validated) -- [~] Create CLI wrapper scripts (SKIPPED - LLMs consume OpenAPI spec directly) -- [~] Remove old shell scripts (DEFERRED - not blocking) - -### Phase 6: Infrastructure Improvements 📋 FUTURE -- [ ] Consolidate Docker network topology into single `docker-dataplane` network - - Currently each stack has its own network (172.22.0.x, 172.25.0.x, 172.20.0.x, etc.) - - Error-prone and unnecessarily complex - - Single shared network simplifies inter-service communication - - Reduces subnet conflicts and improves service discovery - - Update all compose files to use: `networks: [docker-dataplane]` - - Create network once: `docker network create docker-dataplane` - ---- - -## Progress Notes (2025-11-14) - -### Session 1: Foundation & Read Endpoints -**Completed:** -- Created controller and client architecture -- Implemented Portainer client with full CRUD operations for stacks -- Implemented NPM client with JWT refresh and proxy/certificate management -- Built infrastructure controller with 5 read/list endpoints -- Added infrastructure settings to config.py - -**Files Created:** -- `src/controllers/__init__.py` -- `src/controllers/base.py` -- `src/controllers/infrastructure_controller.py` -- `src/clients/__init__.py` -- `src/clients/portainer_client.py` -- `src/clients/npm_client.py` -- `REFACTORING_PLAN.md` (this file) - -### Session 2: Credentials & Testing (2025-11-14 Evening) -**Completed:** -- Created credentials management system (credentials.py gitignored, credentials.example.py committed) -- Updated config.py to import from credentials module with fallback -- Generated Portainer API token programmatically via API -- Integrated infrastructure controller into main.py -- Fixed Pydantic validation bug (status field int→str conversion) -- Tested all read endpoints with live Portainer/NPM infrastructure -- Verified 8 stacks detected, domains with SSL status working - -**Test Results:** -- ✅ GET /infrastructure/health - Portainer connected, NPM accessible -- ✅ GET /infrastructure/services - Returns 8 active stacks -- ✅ GET /infrastructure/services/{name} - Service lookup working -- ✅ GET /infrastructure/domains - Returns proxy hosts with SSL status -- ✅ NPM health check fixed (now accepts 2xx/3xx status codes and follows redirects) - -### Session 3: Write Endpoints (2025-11-14 Evening) -**Completed:** -- Created request/response models for write operations (DeployServiceRequest, UpdateServiceRequest, CreateProxyRequest, OperationResult) -- Implemented POST /infrastructure/services - Deploy new service from compose YAML -- Implemented PUT /infrastructure/services/{name} - Update existing service configuration -- Implemented DELETE /infrastructure/services/{name} - Remove service and stack -- Implemented POST /infrastructure/proxy - Create NPM proxy host with optional SSL certificate -- Updated main.py API description with write endpoints -- Tested all service management endpoints (POST/PUT/DELETE) with live Portainer instance - -**Test Results:** -- ✅ POST /infrastructure/services - Created test-nginx stack (ID: 30) -- ✅ PUT /infrastructure/services/test-nginx - Updated compose with environment variable -- ✅ DELETE /infrastructure/services/test-nginx - Removed stack successfully -- ✅ POST /infrastructure/proxy - Implemented (not tested to avoid production interference) - -**Next Steps:** -1. ~~Refactor existing AI/tools/health endpoints into separate controllers (Phase 4)~~ ✅ DONE (2025-11-14) -2. ~~Fix NPM health check to handle redirects~~ ✅ DONE (2025-11-14) -3. ~~Implement port allocation detection logic~~ ✅ DONE (2025-11-14) -4. ~~Create CLI wrappers for common operations~~ ⊘ SKIPPED (LLMs use OpenAPI) -5. (OPTIONAL) Consolidate Docker networks into `docker-dataplane` (Phase 6) - -### Session 4: NPM Health Check & Port Detection (2025-11-14 Afternoon) -**Completed:** -- Fixed NPM health check to handle redirects properly -- Updated `npm_client.py` to accept 2xx/3xx status codes as healthy -- Enabled explicit redirect following in httpx client -- Verified fix with live NPM instance (now shows 9 proxy hosts) -- Implemented comprehensive port detection in `GET /infrastructure/ports` endpoint -- Added `get_containers()` and `get_container()` methods to PortainerClient -- Enhanced PortInfo model with internal/external hostname and IP fields -- Implemented domain mapping from NPM proxy hosts to services -- Added deduplication logic for port entries (Docker returns duplicates per bind address) - -**Port Detection Features:** -- Scans all running containers across all Portainer endpoints -- Extracts internal port, host port, and protocol for each container -- Maps container names to service names via Docker Compose labels -- Retrieves internal Docker hostnames and IP addresses per network -- Cross-references NPM proxy hosts to identify external domains -- Returns 22 unique port mappings with complete metadata - -**Technical Details:** - -*NPM Health Check:* -- Issue: NPM's `/api` endpoint returns 302 redirect, old code only accepted 200 -- Solution: Accept `200 <= status_code < 400` as healthy response -- Result: NPM health check now returns `true` and proxy hosts are enumerated correctly - -*Port Detection:* -- Queries Portainer Docker API for container list and port mappings -- Extracts NetworkSettings for internal IPs and hostnames -- Builds port→domain map from NPM proxy hosts configuration -- Matches services to external domains using multiple strategies: - - By container name + port - - By internal IP + port - - By host address + host port (localhost, 127.0.0.1, server IP) -- Deduplicates based on (port, container_name, protocol) tuple -- Example output: Nextcloud port 80 → internal IP 172.25.0.3 → external domain cloud.schweitz.net - -### Session 5: Controller Architecture Refactoring (2025-11-14 Evening) -**Completed:** -- Created `ai_controller.py` consolidating chat, models, and conversations endpoints -- Created `tools_controller.py` for web scraper functionality -- Created `health_controller.py` for service health and info endpoints -- Updated `main.py` to use new controller-based architecture -- Removed legacy router imports and inline endpoint definitions -- Tested all refactored endpoints - 16 endpoints working correctly - -**Architecture Changes:** -- All endpoints now follow consistent controller pattern inheriting from `BaseController` -- Controllers use `create_router()` method for FastAPI router configuration -- Clean separation of concerns: - - `ai_controller.py` - AI orchestration and conversation memory (7 endpoints) - - `tools_controller.py` - Utility tools like web scraper (1 endpoint) - - `health_controller.py` - Service status and info (2 endpoints) - - `infrastructure_controller.py` - Infrastructure management (6 endpoints) -- Simplified `main.py` from 220 lines to 152 lines -- Backward compatible - all existing endpoints work identically - -**Test Results:** -- ✅ GET / - Service information -- ✅ GET /health - Health check with Ollama status -- ✅ GET /v1/models - Model listing -- ✅ POST /v1/chat/completions - Chat completions -- ✅ GET /v1/conversations/{id} - Conversation history -- ✅ GET /infrastructure/health - Infrastructure health -- ✅ POST /web-scraper/scrape - Web scraping -- ✅ OpenAPI spec generation - 16 endpoints documented - -## API Authentication Strategy - -### Portainer -- **Method:** Access Token (X-API-Key header) -- **Setup:** Manual creation in UI, store in config/env -- **Duration:** Long-lived -- **Storage:** Environment variable `PORTAINER_API_KEY` - -### Nginx Proxy Manager -- **Method:** JWT Bearer Token -- **Setup:** Login via `/api/tokens` with credentials -- **Duration:** ~24 hours -- **Strategy:** Auto-refresh with stored credentials -- **Storage:** `NPM_EMAIL` and `NPM_PASSWORD` in env - -### Uptime Kuma -- **Method:** Socket.IO WebSocket -- **Setup:** Login via Socket.IO `login` event -- **Duration:** Session-based -- **Strategy:** Maintain persistent connection or re-auth per request -- **Storage:** `KUMA_USERNAME` and `KUMA_PASSWORD` in env - -## Configuration Changes - -### Credentials Management Strategy - -**Use `credentials.py` for sensitive data** (added to `.gitignore`): -- Keeps secrets out of version control -- Easy terminal-based management with editor -- Python format for type safety and autocomplete -- Separate from config for security isolation - -**Implementation:** -1. Create `src/credentials.py` with credentials (gitignored) -2. Create `src/credentials.example.py` as template (committed) -3. Update `config.py` to import from credentials module -4. Add `credentials.py` to `.gitignore` - -**Example `src/credentials.py`:** -```python -""" -Infrastructure credentials (GITIGNORED) -Copy from credentials.example.py and fill in real values -""" - -# Portainer -PORTAINER_URL = "http://localhost:8001" -PORTAINER_API_KEY = "ptr_your_actual_token_here" - -# Nginx Proxy Manager -NPM_URL = "http://localhost:81" -NPM_EMAIL = "jpmschweitzer@gmail.com" -NPM_PASSWORD = "your_actual_password" - -# Uptime Kuma -KUMA_URL = "http://localhost:3001" -KUMA_USERNAME = "admin" -KUMA_PASSWORD = "your_actual_password" -``` - -**Updated `config.py` to use credentials:** -```python -from src.credentials import ( - PORTAINER_URL, PORTAINER_API_KEY, - NPM_URL, NPM_EMAIL, NPM_PASSWORD, - KUMA_URL, KUMA_USERNAME, KUMA_PASSWORD -) - -class Settings(BaseSettings): - # Infrastructure Management (from credentials.py) - portainer_url: str = PORTAINER_URL - portainer_api_key: str = PORTAINER_API_KEY - - npm_url: str = NPM_URL - npm_email: str = NPM_EMAIL - npm_password: str = NPM_PASSWORD - - kuma_url: str = KUMA_URL - kuma_username: str = KUMA_USERNAME - kuma_password: str = KUMA_PASSWORD -``` - -## Benefits - -1. **Cleaner Code:** Separation of concerns, easier to maintain -2. **Automation:** Programmatic service deployment and configuration -3. **Elimination of Shell Scripts:** Replace ad-hoc scripts with proper API -4. **Service Discovery:** Auto-detect running services and configurations -5. **Self-Managing Homelab:** Foundation for autonomous infrastructure - -## Migration Notes - -- Existing `/v1/` endpoints remain unchanged for backward compatibility -- Web scraper endpoints stay at `/web-scraper/` initially -- Old shell scripts in `/stacks/` will be replaced with CLI wrappers diff --git a/services/core-api/create_npm_outpost.py b/services/core-api/create_npm_outpost.py deleted file mode 100644 index d5c6331..0000000 --- a/services/core-api/create_npm_outpost.py +++ /dev/null @@ -1,152 +0,0 @@ -#!/usr/bin/env python3 -""" -Create Authentik Outpost for NPM Forward Authentication - -Creates a dedicated outpost and retrieves its token for deployment. -""" -import asyncio -import sys -sys.path.insert(0, '/home/jpmschweitzer/Projects/portainer-core/services/core-api') - -from src.clients.authentik_client import get_authentik_client - - -async def create_npm_outpost(): - """Create outpost for NPM forward auth and get token""" - authentik = get_authentik_client() - - try: - print("=" * 60) - print("Creating NPM Forward Auth Outpost") - print("=" * 60) - - # Check if provider exists - print("\n1. Checking for proxy provider...") - provider = await authentik.get_provider_by_name_proxy("npm-forward-auth-provider") - - if not provider: - print("❌ Proxy provider not found. Run setup_authentik_forward_auth.py first.") - return False - - provider_id = provider["pk"] - print(f"✓ Found provider (ID: {provider_id})") - - # Check if outpost already exists - print("\n2. Checking for existing NPM outpost...") - try: - existing = await authentik.get_outpost_by_name("npm-forward-auth-outpost") - if existing: - print(f"✓ Outpost already exists (ID: {existing['pk']})") - outpost_id = existing["pk"] - - # Update it to ensure provider is assigned - print("\n3. Updating outpost configuration...") - await authentik.update_outpost( - outpost_id=outpost_id, - providers=[provider_id] - ) - print("✓ Outpost updated with provider") - except Exception: - # Outpost doesn't exist, create it - print("⊘ Outpost doesn't exist, creating new one...") - print("\n3. Creating NPM outpost...") - - outpost = await authentik.create_outpost( - name="npm-forward-auth-outpost", - type="proxy", - providers=[provider_id], - config={ - "authentik_host": "http://192.168.86.149:9000", - "authentik_host_insecure": False, - "log_level": "info", - "docker_labels": None, - "docker_network": None, - "docker_map_ports": True, - "container_image": None, - "kubernetes_replicas": 1, - "kubernetes_namespace": "default" - } - ) - outpost_id = outpost["pk"] - print(f"✓ Created outpost (ID: {outpost_id})") - - # Try to get the service connection token - print("\n4. Retrieving outpost token...") - print("\nNote: Authentik creates service accounts for outposts automatically.") - print("The token format is: ak-outpost--api") - - # Get outpost details to find its service account - outpost_details = await authentik._request("GET", f"outposts/instances/{outpost_id}/") - - print(f"\nOutpost Details:") - print(f" Name: {outpost_details.get('name')}") - print(f" ID: {outpost_details.get('pk')}") - print(f" Type: {outpost_details.get('type')}") - print(f" Providers: {outpost_details.get('providers')}") - - # The outpost service connection details - if 'service_connection' in outpost_details: - print(f" Service Connection: {outpost_details.get('service_connection')}") - - # List tokens to find the one for this outpost - print("\n5. Looking for outpost service token...") - tokens = await authentik.list_tokens() - - outpost_uuid = outpost_details.get('pk') - token_identifier = f"ak-outpost-{outpost_uuid}-api" - - matching_token = None - for token in tokens: - if token.get('identifier') == token_identifier: - matching_token = token - break - - if matching_token: - print(f"✓ Found token: {matching_token.get('identifier')}") - print(f"\n{'=' * 60}") - print("IMPORTANT: Token Key Required") - print("=" * 60) - print("\nAuthentik does not expose token keys via API after creation.") - print("\nTo get the token key:") - print("1. Go to: https://auth.schweitz.net/if/admin/#/core/tokens") - print(f"2. Find token: {token_identifier}") - print("3. Click 'View Token Key' or regenerate the token") - print("4. Copy the token key") - print("\nAlternatively, you can:") - print("1. Delete the existing outpost via UI") - print("2. Create a new outpost via UI") - print("3. Copy the token key when it's displayed") - print("\n" + "=" * 60) - else: - print("\n⚠ No automatic token found.") - print("You may need to manually create a token for the outpost.") - print("\nManual token creation:") - print("1. Go to: https://auth.schweitz.net/if/admin/#/core/tokens") - print("2. Click 'Create'") - print(f"3. Identifier: npm-outpost-token") - print("4. User: Select the outpost service account") - print("5. Intent: API") - print("6. Copy the token key when displayed") - - print("\n" + "=" * 60) - print("Next Steps") - print("=" * 60) - print("\n1. Obtain the outpost token key (see above)") - print("2. Create/update .env.authentik-shared file:") - print(" AUTHENTIK_OUTPOST_TOKEN=") - print("3. Deploy the stack:") - print(" docker-compose -f stacks/authentik-shared.yml --env-file .env.authentik-shared up -d") - print("4. Update NPM hosts to point to port 9001") - - return True - - except Exception as e: - print(f"\n❌ Error: {e}") - import traceback - traceback.print_exc() - return False - - -if __name__ == "__main__": - success = asyncio.run(create_npm_outpost()) - sys.exit(0 if success else 1) diff --git a/services/core-api/setup_authentik_forward_auth.py b/services/core-api/setup_authentik_forward_auth.py deleted file mode 100644 index a0bfb2e..0000000 --- a/services/core-api/setup_authentik_forward_auth.py +++ /dev/null @@ -1,118 +0,0 @@ -#!/usr/bin/env python3 -""" -Setup Authentik Forward Authentication for NPM - -This script creates a proxy provider and outpost for forward authentication -across all NPM-managed domains. -""" -import asyncio -import sys -sys.path.insert(0, '/home/jpmschweitzer/Projects/portainer-core/services/core-api') - -from src.clients.authentik_client import get_authentik_client - - -async def setup_forward_auth(): - """Create proxy provider, application, and outpost for forward auth""" - client = get_authentik_client() - - try: - # Check if Authentik is accessible - print("Checking Authentik connectivity...") - if not await client.health_check(): - print("❌ Authentik is not accessible") - return False - print("✓ Authentik is accessible") - - # Check if proxy provider already exists - print("\nChecking for existing proxy provider...") - existing_provider = await client.get_provider_by_name_proxy("npm-forward-auth-provider") - - if existing_provider: - print(f"✓ Proxy provider already exists (ID: {existing_provider.get('pk')})") - provider = existing_provider - else: - # Create Proxy provider for forward auth - print("\nCreating Proxy provider for forward authentication...") - provider = await client.create_proxy_provider( - name="npm-forward-auth-provider", - external_host="https://auth.schweitz.net", - mode="forward_single", - token_validity=480 # 8 hours - ) - print(f"✓ Created proxy provider (ID: {provider.get('pk')})") - - # Display provider details - print("\n" + "="*60) - print("Proxy Provider Details:") - print("="*60) - print(f"Provider ID: {provider.get('pk')}") - print(f"Mode: {provider.get('mode')}") - print(f"External Host: {provider.get('external_host')}") - print(f"Token Validity: {provider.get('access_token_validity')}") - print(f"Session Duration: {provider.get('session_duration')}") - print("="*60) - - # Check if application already exists - print("\nChecking for existing application...") - existing_app = await client.get_application_by_slug("npm-forward-auth") - - if existing_app: - print(f"✓ Application already exists (slug: {existing_app.get('slug')})") - app = existing_app - else: - # Create application - print("\nCreating application...") - app = await client.create_application( - name="NPM Forward Auth", - slug="npm-forward-auth", - provider_pk=provider.get("pk"), - launch_url="https://auth.schweitz.net" - ) - print(f"✓ Created application (slug: {app.get('slug')})") - - # Check if outpost already exists - print("\nChecking for existing outpost...") - existing_outpost = await client.get_outpost_by_name("npm-forward-auth-outpost") - - if existing_outpost: - print(f"✓ Outpost already exists (ID: {existing_outpost.get('pk')})") - outpost = existing_outpost - else: - # Create outpost - print("\nCreating outpost...") - outpost = await client.create_outpost( - name="npm-forward-auth-outpost", - type="proxy", - providers=[provider.get("pk")], - config={ - "authentik_host": "https://auth.schweitz.net", - "authentik_host_insecure": False, - "log_level": "info" - } - ) - print(f"✓ Created outpost (ID: {outpost.get('pk')})") - - print("\n" + "="*60) - print("Setup Complete!") - print("="*60) - print("\nNext steps:") - print("1. Deploy the Authentik outpost container") - print("2. Configure NPM proxy hosts with forward auth") - print("3. Test the SSO flow") - print("\nOutpost deployment command will be generated...") - - return True - - except Exception as e: - print(f"\n❌ Error: {e}") - import traceback - traceback.print_exc() - return False - finally: - await client.close() - - -if __name__ == "__main__": - success = asyncio.run(setup_forward_auth()) - sys.exit(0 if success else 1) diff --git a/services/core-api/src/auth/oidc.py b/services/core-api/src/auth/oidc.py index d8334bf..f381c13 100644 --- a/services/core-api/src/auth/oidc.py +++ b/services/core-api/src/auth/oidc.py @@ -4,7 +4,7 @@ OIDC Authentication Module Provides OAuth2/OIDC token validation for FastAPI using Authentik as IdP. Implements bearer token authentication with JWT verification. """ -from fastapi import Depends, HTTPException, Security +from fastapi import Depends, HTTPException, Security, Request from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials from jose import jwt, JWTError import httpx @@ -233,3 +233,104 @@ async def get_optional_user( except HTTPException: # Invalid token - return None instead of raising return None + + +async def get_forward_auth_user( + request: Request +) -> Optional[Dict]: + """ + Authentik Forward Auth authentication for external access via NPM + + This dependency allows: + - External access through api.schweitz.net (with Authentik forward auth headers) - REQUIRES authentication + - Internal direct access (no forward auth headers) - ALLOWED without authentication + + When accessing through NPM with Authentik forward auth enabled, NPM adds headers like: + - X-authentik-username + - X-authentik-email + - X-authentik-groups + - X-authentik-name + - X-authentik-uid + + Args: + request: FastAPI request object containing headers + + Returns: + User info dict if authenticated via forward auth headers + None if accessed internally (no forward auth headers) + + Raises: + HTTPException 401: If forward auth headers present but invalid/incomplete + """ + # Check for Authentik forward auth headers + username = request.headers.get("x-authentik-username") + email = request.headers.get("x-authentik-email") + groups = request.headers.get("x-authentik-groups") + name = request.headers.get("x-authentik-name") + uid = request.headers.get("x-authentik-uid") + + # If NO forward auth headers present, this is internal access - allow it + if not username and not email: + logger.debug("No forward auth headers - allowing internal access") + return None + + # Forward auth headers present (external access via api.schweitz.net) + # Validate authentication + if not username or not email: + logger.warning("Incomplete forward auth headers detected") + raise HTTPException( + status_code=401, + detail="Authentication required - incomplete forward auth headers" + ) + + # Parse groups (comma-separated string to list) + groups_list = [g.strip() for g in groups.split(",")] if groups else [] + + user_info = { + "username": username, + "email": email, + "name": name or username, + "groups": groups_list, + "uid": uid, + "auth_method": "forward_auth" + } + + logger.info(f"Authenticated via forward auth: {email} (groups: {groups_list})") + return user_info + + +async def get_forward_auth_admin( + user: Optional[Dict] = Depends(get_forward_auth_user) +) -> Dict: + """ + Require admin access for external requests, allow all internal requests + + Use this dependency for endpoints that require admin access when accessed + externally through api.schweitz.net, but allow unrestricted internal access. + + Args: + user: User info from get_forward_auth_user + + Returns: + User info dict if user is admin or if accessed internally + + Raises: + HTTPException 403: If external user is not in admin/authentik Admins group + """ + # Internal access (no forward auth headers) - allow all + if user is None: + logger.debug("Internal access - allowing without admin check") + return {"email": "internal", "groups": ["admin"], "auth_method": "internal"} + + # External access - check admin group membership + groups = user.get("groups", []) + + if "admin" not in groups and "authentik Admins" not in groups: + user_email = user.get("email", "unknown") + logger.warning(f"User {user_email} attempted admin access (groups: {groups})") + raise HTTPException( + status_code=403, + detail="Admin access required" + ) + + return user diff --git a/services/core-api/src/clients/kuma_client.py b/services/core-api/src/clients/kuma_client.py index f85484d..6069438 100644 --- a/services/core-api/src/clients/kuma_client.py +++ b/services/core-api/src/clients/kuma_client.py @@ -2,9 +2,12 @@ Uptime Kuma Socket.IO Client Provides interface to Uptime Kuma via Socket.IO for monitor management. +Also provides metrics API access for real-time status data. """ import socketio import asyncio +import httpx +import re from typing import Optional, Dict, List, Any from src.logging_config import get_logger from src.config import get_settings @@ -125,31 +128,76 @@ class KumaClient: async def get_monitors(self) -> List[Dict[str, Any]]: """ - List all monitors + List all monitors with uptime data Returns: - List of monitor configurations + List of monitor configurations with uptime_24h field """ await self._ensure_connected() try: - # Get monitor list - response = await self.sio.call('getMonitorList', timeout=self.timeout) + # Storage for monitor list and uptime data received via events + monitor_list_data = {} + uptime_list_data = {} + monitor_event_received = asyncio.Event() + uptime_event_received = asyncio.Event() - if response and isinstance(response, dict): - # Uptime Kuma returns monitors as a dict with monitor IDs as keys + # Register event handler for monitorList + @self.sio.event + async def monitorList(data): + nonlocal monitor_list_data + monitor_list_data = data + monitor_event_received.set() + + # Register event handler for uptimeList (24h uptime percentages) + @self.sio.event + async def uptimeList(monitor_id, uptime_data): + nonlocal uptime_list_data + # uptime_data is typically a dict with time periods: {"24": 99.5, "720": 98.2, ...} + uptime_list_data[str(monitor_id)] = uptime_data + # Don't set event here as we'll get multiple calls + + # Request monitor list - this triggers the server to send monitorList event + response = await self.sio.call('getMonitorList', timeout=self.timeout) + logger.info(f"getMonitorList call response: {response}") + + # Wait for the monitorList event (with timeout) + try: + await asyncio.wait_for(monitor_event_received.wait(), timeout=5.0) + logger.info(f"Received monitorList event with {len(monitor_list_data)} items") + + # Give time for uptimeList events to arrive + await asyncio.sleep(0.5) + logger.info(f"Received uptime data for {len(uptime_list_data)} monitors") + except asyncio.TimeoutError: + logger.warning("Timeout waiting for monitorList event") + + # Process the monitor list data + if monitor_list_data and isinstance(monitor_list_data, dict): monitors = [] - for monitor_id, monitor_data in response.items(): + for monitor_id, monitor_data in monitor_list_data.items(): if isinstance(monitor_data, dict): monitor_data['id'] = int(monitor_id) + + # Add uptime data if available + uptime_info = uptime_list_data.get(str(monitor_id), {}) + if isinstance(uptime_info, dict): + # Uptime Kuma provides 24h uptime as key "24" + monitor_data['uptime_24h'] = float(uptime_info.get('24', 0)) + else: + monitor_data['uptime_24h'] = 0.0 + monitors.append(monitor_data) self._monitors_cache[int(monitor_id)] = monitor_data + logger.info(f"Found {len(monitors)} monitors total") return monitors + + logger.warning(f"No valid monitor data received") return [] except Exception as e: - logger.error(f"Failed to get monitors: {e}") + logger.error(f"Failed to get monitors: {e}", exc_info=True) raise async def get_monitor(self, monitor_id: int) -> Dict[str, Any]: @@ -419,6 +467,78 @@ class KumaClient: await self.delete_monitor(monitor["id"]) return True + async def get_metrics_status(self) -> Dict[str, Dict[str, Any]]: + """ + Get monitor status from Prometheus metrics endpoint + + This is simpler and more reliable than Socket.IO for getting current status. + Returns real-time UP/DOWN status but not historical uptime percentages. + + Returns: + Dict mapping monitor names to status info: + { + "Portainer": { + "status": 1, # 1=UP, 0=DOWN, 2=PENDING, 3=MAINTENANCE + "response_time": 5, # ms + "monitor_type": "http", + "url": "http://192.168.86.149:8001" + }, + ... + } + """ + try: + # Use API key authentication + api_key = settings.kuma_api_key + if not api_key: + logger.warning("Kuma API key not configured") + return {} + + # Fetch metrics with HTTP Basic Auth (empty username, API key as password) + async with httpx.AsyncClient(timeout=10.0) as client: + response = await client.get( + f"{self.base_url}/metrics", + auth=("", api_key) + ) + response.raise_for_status() + metrics_text = response.text + + # Parse Prometheus format metrics + # Format: metric_name{label1="value1",label2="value2"} value + monitor_data = {} + + # Parse monitor_status lines + status_pattern = r'monitor_status\{monitor_name="([^"]+)",.*?\} (\d+)' + for match in re.finditer(status_pattern, metrics_text): + monitor_name = match.group(1) + status = int(match.group(2)) + + if monitor_name not in monitor_data: + monitor_data[monitor_name] = {} + monitor_data[monitor_name]['status'] = status + + # Parse monitor_response_time lines + response_pattern = r'monitor_response_time\{monitor_name="([^"]+)",monitor_type="([^"]+)",monitor_url="([^"]+)",.*?\} ([\d.]+)' + for match in re.finditer(response_pattern, metrics_text): + monitor_name = match.group(1) + monitor_type = match.group(2) + monitor_url = match.group(3) + response_time = float(match.group(4)) + + if monitor_name not in monitor_data: + monitor_data[monitor_name] = {} + monitor_data[monitor_name].update({ + 'response_time': response_time, + 'monitor_type': monitor_type, + 'url': monitor_url + }) + + logger.info(f"Fetched metrics for {len(monitor_data)} monitors") + return monitor_data + + except Exception as e: + logger.error(f"Failed to fetch metrics: {e}") + return {} + async def __aenter__(self): """Async context manager entry""" await self._ensure_connected() diff --git a/services/core-api/src/config.py b/services/core-api/src/config.py index f0d518f..07c10d4 100644 --- a/services/core-api/src/config.py +++ b/services/core-api/src/config.py @@ -9,7 +9,7 @@ try: from src.credentials import ( PORTAINER_URL, PORTAINER_API_KEY, NPM_URL, NPM_EMAIL, NPM_PASSWORD, - KUMA_URL, KUMA_USERNAME, KUMA_PASSWORD + KUMA_URL, KUMA_USERNAME, KUMA_PASSWORD, KUMA_API_KEY ) except ImportError: # Fallback to empty strings if credentials.py doesn't exist @@ -22,6 +22,7 @@ except ImportError: KUMA_URL = "http://localhost:3001" KUMA_USERNAME = "" KUMA_PASSWORD = "" + KUMA_API_KEY = "" class Settings(BaseSettings): @@ -43,7 +44,7 @@ class Settings(BaseSettings): cors_headers: list[str] = ["*"] # Logging - log_level: str = "INFO" + log_level: str = "DEBUG" # Ollama Configuration (for AI orchestration) ollama_base_url: str = "http://ollama:11434" @@ -88,6 +89,7 @@ class Settings(BaseSettings): kuma_url: str = KUMA_URL kuma_username: str = KUMA_USERNAME kuma_password: str = KUMA_PASSWORD + kuma_api_key: str = KUMA_API_KEY # OIDC Authentication (Authentik) oidc_enabled: bool = False # Set to True to require authentication diff --git a/services/core-api/src/controllers/infrastructure_controller.py b/services/core-api/src/controllers/infrastructure_controller.py index e1cf553..6295e56 100644 --- a/services/core-api/src/controllers/infrastructure_controller.py +++ b/services/core-api/src/controllers/infrastructure_controller.py @@ -14,7 +14,7 @@ from src.clients.npm_client import get_npm_client from src.clients.kuma_client import get_kuma_client from src.logging_config import get_logger from src import service_groups -from src.auth.oidc import get_admin_user +from src.auth.oidc import get_admin_user, get_forward_auth_admin logger = get_logger(__name__) @@ -751,11 +751,11 @@ class InfrastructureController(BaseController): "/services/{name}/stop", response_model=OperationResult, summary="Stop a service or service group", - description="Stop a service or service group by pausing monitors and stopping containers. Requires admin authentication." + description="Stop a service or service group by pausing monitors and stopping containers. Requires admin authentication when accessed externally via api.schweitz.net." ) async def stop_service( name: str, - user: Dict = Depends(get_admin_user) + user: Dict = Depends(get_forward_auth_admin) ): """ Stop a service or service group @@ -866,11 +866,11 @@ class InfrastructureController(BaseController): "/services/{name}/start", response_model=OperationResult, summary="Start a service or service group", - description="Start a service or service group by starting containers and resuming monitors. Requires admin authentication." + description="Start a service or service group by starting containers and resuming monitors. Requires admin authentication when accessed externally via api.schweitz.net." ) async def start_service( name: str, - user: Dict = Depends(get_admin_user) + user: Dict = Depends(get_forward_auth_admin) ): """ Start a service or service group @@ -971,6 +971,118 @@ class InfrastructureController(BaseController): # ===== Monitoring Endpoints ===== + @router.get( + "/widget-data", + summary="Get combined data for service control widget", + response_model=Dict[str, Any] + ) + async def get_widget_data(): + """ + Get combined service and monitor data for the widget + + Returns all data needed by service-control widget in a single call: + - Service list with status and container counts + - Monitor list with uptime percentages + - Service groups and always-on list + + This endpoint is designed for browser-based widgets to avoid + multiple API calls and cross-origin issues. + """ + try: + portainer = get_portainer_client() + kuma = get_kuma_client() + npm = get_npm_client() + + # Fetch services (same logic as /services endpoint) + stacks = await portainer.get_stacks() + proxy_hosts = await npm.get_proxy_hosts() + + # Build domain mapping + domain_map = {} + for proxy in proxy_hosts: + for domain in proxy.get("domain_names", []): + forward_host = proxy.get("forward_host", "") + domain_map[domain] = forward_host + + services = [] + for stack in stacks: + stack_name = stack.get("Name", "") + endpoint_id = stack.get("EndpointId") + domains = [ + domain for domain, host in domain_map.items() + if stack_name in host or host in stack_name + ] + + # Get container status + containers_running = 0 + containers_total = 0 + try: + all_containers = await portainer.get_containers(endpoint_id, all_containers=True) + for container in all_containers: + labels = container.get("Labels", {}) + container_stack = labels.get("com.docker.compose.project", "") + + if container_stack.lower() == stack_name.lower(): + containers_total += 1 + if container.get("State", "") == "running": + containers_running += 1 + except Exception as e: + logger.warning(f"Failed to get container status for {stack_name}: {e}") + + services.append({ + "name": stack_name, + "stack_id": stack.get("Id"), + "status": "active" if stack.get("Status") == 1 else "inactive", + "endpoint_id": endpoint_id, + "domains": domains, + "running": containers_running > 0, + "containers_running": containers_running, + "containers_total": containers_total + }) + + # Fetch monitors with real-time status from metrics endpoint + monitors_list = [] + try: + # Get real-time status from Prometheus metrics + metrics_data = await kuma.get_metrics_status() + + for monitor_name, monitor_info in metrics_data.items(): + # Status: 1=UP, 0=DOWN, 2=PENDING, 3=MAINTENANCE + status = monitor_info.get('status', 0) + + # Convert status to simple up/down for widget + # Treat UP (1) as 100%, anything else as 0% + status_percentage = 100.0 if status == 1 else 0.0 + + monitors_list.append({ + "id": None, # Not available from metrics + "name": monitor_name, + "uptime_24h": status_percentage, # Current status as percentage + "active": True, # Assume active if in metrics + "status": status, # 1=UP, 0=DOWN, 2=PENDING, 3=MAINTENANCE + "response_time": monitor_info.get('response_time', 0) + }) + + logger.info(f"Fetched status for {len(monitors_list)} monitors from metrics") + except Exception as e: + logger.warning(f"Failed to fetch monitors: {e}") + # Continue without monitor data rather than failing + + return { + "success": True, + "services": services, + "monitors": monitors_list, + "service_groups": { + "groups": service_groups.list_service_groups(), + "always_on": list(service_groups.ALWAYS_ON_SERVICES), + "stoppable": service_groups.list_stoppable_services() + } + } + + except Exception as e: + logger.error(f"Failed to fetch widget data: {e}") + raise HTTPException(status_code=500, detail=f"Failed to fetch widget data: {str(e)}") + @router.get( "/monitors", summary="List all monitors", diff --git a/services/core-api/src/service_groups.py b/services/core-api/src/service_groups.py index 9177699..55512a4 100644 --- a/services/core-api/src/service_groups.py +++ b/services/core-api/src/service_groups.py @@ -16,6 +16,9 @@ ALWAYS_ON_SERVICES: Set[str] = { "watchtower", "netdata", "maintenance", + "postgres-shared", + "redis-shared", + "authentik", } # Service groups - services that should be started/stopped together @@ -25,8 +28,6 @@ SERVICE_GROUPS: Dict[str, List[str]] = { ], "nextcloud": [ "nextcloud", - "nextcloud-db", - "nextcloud-redis", ], "gitea": [ "gitea", diff --git a/services/core-api/static/widgets/service-control.html b/services/core-api/static/widgets/service-control.html index 722f7e5..ec2a350 100644 --- a/services/core-api/static/widgets/service-control.html +++ b/services/core-api/static/widgets/service-control.html @@ -15,106 +15,163 @@ font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif; background: transparent; color: #e0e0e0; - padding: 10px; + padding: 15px; } .container { - max-width: 1200px; - margin: 0 auto; + max-width: 100%; } - h2 { + .section { + margin-bottom: 30px; + } + + .section-header { color: #fff; - margin-bottom: 15px; - font-size: 20px; - font-weight: 500; + font-size: 18px; + font-weight: 600; + margin-bottom: 12px; + padding-bottom: 8px; + border-bottom: 2px solid rgba(255, 255, 255, 0.1); } - .service-grid { - display: grid; - grid-template-columns: repeat(auto-fill, minmax(300px, 1fr)); + .service-list { + display: flex; + flex-direction: column; + gap: 8px; + } + + .service-row { + background: rgba(40, 40, 40, 0.95); + border: 1px solid rgba(255, 255, 255, 0.1); + border-radius: 6px; + padding: 12px 16px; + display: flex; + align-items: center; + justify-content: space-between; + transition: all 0.2s ease; gap: 15px; } - .service-card { - background: rgba(40, 40, 40, 0.95); - border: 1px solid rgba(255, 255, 255, 0.1); - border-radius: 8px; - padding: 15px; - transition: all 0.3s ease; + .service-row:hover { + border-color: rgba(66, 153, 225, 0.4); + background: rgba(45, 45, 45, 0.95); } - .service-card:hover { - border-color: rgba(66, 153, 225, 0.5); - box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3); - } - - .service-header { + .service-left { display: flex; - justify-content: space-between; align-items: center; - margin-bottom: 12px; + gap: 15px; + flex: 1; + min-width: 0; } .service-name { - font-size: 16px; + font-size: 15px; font-weight: 600; color: #fff; text-transform: capitalize; + min-width: 300px; } - .status-badge { - padding: 4px 12px; - border-radius: 12px; - font-size: 12px; - font-weight: 600; - text-transform: uppercase; + .service-status { + display: flex; + align-items: center; + gap: 8px; + min-width: 120px; } - .status-running { - background: rgba(72, 187, 120, 0.2); - color: #48bb78; - border: 1px solid rgba(72, 187, 120, 0.4); + .status-indicator { + width: 8px; + height: 8px; + border-radius: 50%; + flex-shrink: 0; } - .status-stopped { - background: rgba(245, 101, 101, 0.2); - color: #f56565; - border: 1px solid rgba(245, 101, 101, 0.4); + .status-indicator.running { + background: #48bb78; + box-shadow: 0 0 8px rgba(72, 187, 120, 0.6); } - .status-loading { - background: rgba(237, 137, 54, 0.2); - color: #ed8936; - border: 1px solid rgba(237, 137, 54, 0.4); + .status-indicator.stopped { + background: #f56565; + box-shadow: 0 0 8px rgba(245, 101, 101, 0.6); } - .service-info { + .status-text { font-size: 13px; color: #a0a0a0; - margin-bottom: 12px; } - .service-actions { + .uptime-status { display: flex; + align-items: center; + gap: 8px; + min-width: 150px; + padding: 4px 10px; + background: rgba(0, 0, 0, 0.2); + border-radius: 4px; + cursor: pointer; + transition: background 0.2s; + text-decoration: none; + color: inherit; + } + + .uptime-status:hover { + background: rgba(0, 0, 0, 0.4); + } + + .uptime-percentage { + font-size: 13px; + font-weight: 600; + } + + .uptime-percentage.excellent { + color: #48bb78; + } + + .uptime-percentage.good { + color: #68d391; + } + + .uptime-percentage.warning { + color: #ed8936; + } + + .uptime-percentage.critical { + color: #f56565; + } + + .uptime-percentage.unknown { + color: #718096; + } + + .uptime-icon { + font-size: 11px; + color: #a0a0a0; + } + + .service-right { + display: flex; + align-items: center; gap: 8px; } .btn { - flex: 1; - padding: 8px 12px; + padding: 6px 16px; border: none; - border-radius: 6px; - font-size: 13px; + border-radius: 4px; + font-size: 12px; font-weight: 600; cursor: pointer; transition: all 0.2s ease; text-transform: uppercase; letter-spacing: 0.5px; + min-width: 70px; } .btn:disabled { - opacity: 0.5; + opacity: 0.3; cursor: not-allowed; } @@ -126,6 +183,7 @@ .btn-start:hover:not(:disabled) { background: linear-gradient(135deg, #38a169 0%, #2f855a 100%); transform: translateY(-1px); + box-shadow: 0 2px 8px rgba(72, 187, 120, 0.3); } .btn-stop { @@ -136,21 +194,12 @@ .btn-stop:hover:not(:disabled) { background: linear-gradient(135deg, #e53e3e 0%, #c53030 100%); transform: translateY(-1px); - } - - .btn-restart { - background: linear-gradient(135deg, #4299e1 0%, #3182ce 100%); - color: white; - } - - .btn-restart:hover:not(:disabled) { - background: linear-gradient(135deg, #3182ce 0%, #2c5282 100%); - transform: translateY(-1px); + box-shadow: 0 2px 8px rgba(245, 101, 101, 0.3); } .loading { text-align: center; - padding: 40px; + padding: 30px; color: #a0a0a0; } @@ -164,14 +213,14 @@ } .always-on-badge { - display: inline-block; - padding: 2px 8px; - background: rgba(66, 153, 225, 0.2); + font-size: 10px; color: #4299e1; - border: 1px solid rgba(66, 153, 225, 0.4); - border-radius: 10px; - font-size: 11px; + background: rgba(66, 153, 225, 0.15); + padding: 2px 6px; + border-radius: 3px; margin-left: 8px; + text-transform: uppercase; + font-weight: 600; } @keyframes spin { @@ -180,45 +229,102 @@ .spinner { display: inline-block; - width: 14px; - height: 14px; + width: 12px; + height: 12px; border: 2px solid rgba(255, 255, 255, 0.3); border-top-color: #fff; border-radius: 50%; animation: spin 0.6s linear infinite; margin-right: 6px; } + + /* Responsive design */ + @media (max-width: 768px) { + .service-row { + flex-wrap: wrap; + } + + .service-name { + min-width: 100px; + } + + .uptime-status { + min-width: 100px; + } + + .service-right { + width: 100%; + justify-content: flex-end; + } + }
-

🎛️ On-Demand Services

-
Loading services...
+ +
+
🎛️ Stoppable Services
+
Loading services...
+
+ +
+
🔒 Always-On Infrastructure
+
Loading infrastructure...
+
diff --git a/services/core-api/test_headers.py b/services/core-api/test_headers.py deleted file mode 100644 index e0e7b49..0000000 --- a/services/core-api/test_headers.py +++ /dev/null @@ -1,54 +0,0 @@ -#!/usr/bin/env python3 -""" -Test what headers are being sent to Organizr -""" -import asyncio -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).parent)) - -from src.clients.npm_client import get_npm_client - - -async def main(): - npm = get_npm_client() - - # Find home.schweitz.net - hosts = await npm.get_proxy_hosts() - - for host in hosts: - if 'home.schweitz.net' in host.get('domain_names', []): - print(f"Found: {', '.join(host.get('domain_names', []))}") - print(f"Forward to: {host.get('forward_scheme')}://{host.get('forward_host')}:{host.get('forward_port')}") - print() - - config = host.get('advanced_config', '') - - print("Checking for Authentik headers in nginx config:") - print("=" * 60) - - headers_to_check = [ - 'X-authentik-username', - 'X-authentik-email', - 'X-authentik-groups', - 'X-authentik-name', - 'X-authentik-uid' - ] - - for header in headers_to_check: - if f'proxy_set_header {header}' in config: - print(f"✓ {header} is configured") - else: - print(f"✗ {header} is NOT configured") - - print() - print("Full advanced config:") - print("=" * 60) - print(config) - - break - - -if __name__ == "__main__": - asyncio.run(main()) diff --git a/services/core-api/update_npm_to_port_9001.py b/services/core-api/update_npm_to_port_9001.py deleted file mode 100644 index 2d75924..0000000 --- a/services/core-api/update_npm_to_port_9001.py +++ /dev/null @@ -1,115 +0,0 @@ -#!/usr/bin/env python3 -""" -Update all NPM proxy hosts to use port 9001 for Authentik forward auth -""" -import asyncio -import httpx -import os - -NPM_URL = os.getenv("NPM_URL", "http://192.168.86.149:81") -NPM_EMAIL = os.getenv("NPM_EMAIL", "admin@example.com") -NPM_PASSWORD = os.getenv("NPM_PASSWORD", "changeme") - - -async def get_npm_token(): - """Get NPM authentication token""" - async with httpx.AsyncClient() as client: - response = await client.post( - f"{NPM_URL}/api/tokens", - json={"identity": NPM_EMAIL, "secret": NPM_PASSWORD} - ) - response.raise_for_status() - return response.json()["token"] - - -async def get_proxy_hosts(token): - """Get all proxy hosts""" - headers = {"Authorization": f"Bearer {token}"} - async with httpx.AsyncClient() as client: - response = await client.get( - f"{NPM_URL}/api/nginx/proxy-hosts", - headers=headers - ) - response.raise_for_status() - return response.json() - - -async def update_proxy_host(token, host_id, config): - """Update a proxy host""" - headers = {"Authorization": f"Bearer {token}"} - async with httpx.AsyncClient() as client: - response = await client.put( - f"{NPM_URL}/api/nginx/proxy-hosts/{host_id}", - headers=headers, - json=config - ) - response.raise_for_status() - return response.json() - - -async def main(): - print("Updating NPM proxy hosts to use port 9001...\n") - - # Get token - token = await get_npm_token() - - # Get all proxy hosts - hosts = await get_proxy_hosts(token) - - updated = [] - skipped = [] - - for host in hosts: - host_id = host.get("id") - domain_names = host.get("domain_names", []) - domain_str = ", ".join(domain_names) - advanced_config = host.get("advanced_config", "") - - # Skip if no authentik config - if "authentik" not in advanced_config.lower(): - continue - - # Skip if already port 9001 - if ":9001" in advanced_config: - print(f"⊘ {domain_str} - Already using port 9001") - skipped.append(domain_str) - continue - - # Update 9000 to 9001 - if ":9000" in advanced_config: - print(f"⟳ {domain_str} - Updating to port 9001...", end=" ") - - new_config = advanced_config.replace(":9000", ":9001") - - # Clean config - readonly_fields = [ - "id", "created_on", "modified_on", "owner", "owner_user_id", - "certificate", "use_default_location", "ipv6", "meta", - "nginx_online", "nginx_err", "access_list", "certificate_id" - ] - - clean_host = {k: v for k, v in host.items() if k not in readonly_fields} - clean_host["advanced_config"] = new_config - - if "locations" not in clean_host or clean_host["locations"] is None: - clean_host["locations"] = [] - - try: - await update_proxy_host(token, host_id, clean_host) - print("✓") - updated.append(domain_str) - except Exception as e: - print(f"✗ Error: {e}") - - print(f"\n{'='*60}") - print(f"Updated: {len(updated)} hosts") - print(f"Skipped: {len(skipped)} hosts") - - if updated: - print("\nUpdated hosts:") - for d in updated: - print(f" • {d}") - - -if __name__ == "__main__": - asyncio.run(main()) diff --git a/stacks/authentik-shared.yml b/stacks/authentik-shared.yml deleted file mode 100644 index 9079edb..0000000 --- a/stacks/authentik-shared.yml +++ /dev/null @@ -1,194 +0,0 @@ -version: '3.8' - -# Authentik - Identity Provider for SSO (Using Shared Infrastructure) -# Phase 1: Foundation - Google OAuth Integration -# Ports: 9000 (HTTP), 9443 (HTTPS) -# GPU: No -# Dependencies: postgres-shared, redis-shared - -services: - authentik-server: - image: ghcr.io/goauthentik/server:latest - container_name: authentik-server - restart: unless-stopped - command: server - ports: - - "9000:9000" - # Port 9443 removed - use NPM for HTTPS termination - environment: - # Database configuration (shared PostgreSQL) - AUTHENTIK_POSTGRESQL__HOST: postgres-shared - AUTHENTIK_POSTGRESQL__PORT: 5432 - AUTHENTIK_POSTGRESQL__NAME: authentik - AUTHENTIK_POSTGRESQL__USER: authentik_user - AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_DB_PASSWORD:?database password required} - - # Cache configuration (shared Redis, database 1) - AUTHENTIK_REDIS__HOST: redis-shared - AUTHENTIK_REDIS__PORT: 6379 - AUTHENTIK_REDIS__DB: 1 - - # Authentik secret key - AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:?secret key required} - - # Error reporting (disabled) - AUTHENTIK_ERROR_REPORTING__ENABLED: "false" - - # Performance tuning for home use - WORKERS: 2 - - # Email configuration (optional - configure later if needed) - # AUTHENTIK_EMAIL__HOST: smtp.gmail.com - # AUTHENTIK_EMAIL__PORT: 587 - # AUTHENTIK_EMAIL__USERNAME: your-email@gmail.com - # AUTHENTIK_EMAIL__PASSWORD: your-app-password - # AUTHENTIK_EMAIL__USE_TLS: "true" - # AUTHENTIK_EMAIL__FROM: authentik@schweitz.net - - # Timezone - TZ: Europe/Amsterdam - volumes: - - /home/jpmschweitzer/docker-data/authentik/media:/media - - /home/jpmschweitzer/docker-data/authentik/custom-templates:/templates - networks: - - docker-dataplane - depends_on: - - postgres-shared - - redis-shared - deploy: - resources: - limits: - memory: 256M - - authentik-worker: - image: ghcr.io/goauthentik/server:latest - container_name: authentik-worker - restart: unless-stopped - command: worker - environment: - # Database configuration (shared PostgreSQL) - AUTHENTIK_POSTGRESQL__HOST: postgres-shared - AUTHENTIK_POSTGRESQL__PORT: 5432 - AUTHENTIK_POSTGRESQL__NAME: authentik - AUTHENTIK_POSTGRESQL__USER: authentik_user - AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_DB_PASSWORD} - - # Cache configuration (shared Redis, database 1) - AUTHENTIK_REDIS__HOST: redis-shared - AUTHENTIK_REDIS__PORT: 6379 - AUTHENTIK_REDIS__DB: 1 - - # Authentik secret key - AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY} - - # Error reporting (disabled) - AUTHENTIK_ERROR_REPORTING__ENABLED: "false" - - # Timezone - TZ: Europe/Amsterdam - user: root - volumes: - - /var/run/docker.sock:/var/run/docker.sock - - /home/jpmschweitzer/docker-data/authentik/media:/media - - /home/jpmschweitzer/docker-data/authentik/certs:/certs - - /home/jpmschweitzer/docker-data/authentik/custom-templates:/templates - networks: - - docker-dataplane - depends_on: - - postgres-shared - - redis-shared - deploy: - resources: - limits: - memory: 192M - - authentik-proxy-outpost: - image: ghcr.io/goauthentik/proxy:latest - container_name: authentik-proxy-outpost - restart: unless-stopped - network_mode: host - environment: - # Authentik connection - AUTHENTIK_HOST: http://192.168.86.149:9000 - AUTHENTIK_INSECURE: "false" - AUTHENTIK_TOKEN: ${AUTHENTIK_OUTPOST_TOKEN:?outpost token required} - - # Logging - AUTHENTIK_LOG_LEVEL: info - - # Port configuration - AUTHENTIK_LISTEN__HTTP: 0.0.0.0:9001 - AUTHENTIK_LISTEN__METRICS: 0.0.0.0:9300 - depends_on: - - authentik-server - labels: - - "com.centurylinklabs.watchtower.enable=true" - healthcheck: - test: ["CMD", "wget", "--spider", "-q", "http://localhost:9001/outpost.goauthentik.io/ping"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 30s - -networks: - docker-dataplane: - external: true - name: docker-dataplane - -# Prerequisites: -# -# 1. Deploy shared-infrastructure stack first! -# docker-compose -f shared-infrastructure.yml up -d -# -# 2. Verify shared services are running: -# docker ps | grep -E 'postgres-shared|redis-shared' -# -# 3. Create Authentik data directories (if not exists): -# mkdir -p ~/docker-data/authentik/{media,certs,custom-templates} -# -# 4. Create .env file with: -# AUTHENTIK_DB_PASSWORD= -# AUTHENTIK_SECRET_KEY= -# -# 5. Deploy this stack: -# docker-compose -f authentik-shared.yml --env-file .env.authentik-shared up -d -# -# After Deployment: -# -# 1. Wait for containers to start (may take 30-60 seconds for DB migrations) -# -# 2. Check logs: -# docker logs authentik-server -# docker logs authentik-worker -# -# 3. Access initial setup: http://localhost:9000/if/flow/initial-setup/ -# - Create admin account (akadmin recommended) -# - Set strong password -# -# 4. Configure NPM reverse proxy: -# - Domain: auth.schweitz.net -# - Forward to: authentik-server:9000 -# - SSL: Let's Encrypt -# - Websockets: Enabled -# -# 5. Access admin interface: https://auth.schweitz.net/if/admin/ -# -# Connection Details: -# -# Database: -# - Host: postgres-shared (from containers) / localhost (from host) -# - Port: 5432 -# - Database: authentik -# - User: authentik_user -# -# Cache: -# - Host: redis-shared (from containers) / localhost (from host) -# - Port: 6379 -# - Database: 1 -# -# Resource Usage (optimized for home use): -# - Server: 256MB RAM limit (WORKERS=2 reduces Gunicorn processes) -# - Worker: 192MB RAM limit -# - Proxy Outpost: ~32MB RAM -# - Total Authentik: ~480MB max (vs ~700MB with dedicated PostgreSQL/Redis) -# - Savings: ~400MB by using shared infrastructure! diff --git a/stacks/core-api.yml b/stacks/core-api.yml index 3645bb5..9a2e59d 100644 --- a/stacks/core-api.yml +++ b/stacks/core-api.yml @@ -29,7 +29,7 @@ services: /venv/bin/uvicorn src.main:app --host 0.0.0.0 --port 8083 - --workers 2 + --workers 1 " ports: diff --git a/stacks/nextcloud-db-consolidation.md b/stacks/nextcloud-db-consolidation.md deleted file mode 100644 index a9aae48..0000000 --- a/stacks/nextcloud-db-consolidation.md +++ /dev/null @@ -1,398 +0,0 @@ -# Nextcloud Database Consolidation Plan - -**Goal**: Fresh Nextcloud installation using shared PostgreSQL + Redis infrastructure - -**Date**: 2025-11-16 -**Status**: APPROVED - Complete wipe and fresh start -**Approach**: No migration, no backups - complete fresh installation - ---- - -## Current State - -### Existing Setup -```yaml -nextcloud-db (MariaDB 10.11) - ├─ Database: nextcloud - ├─ User: nextcloud - ├─ Data: /home/jpmschweitzer/docker-data/nextcloud/db - └─ Network: docker-dataplane - -nextcloud-redis (Redis Alpine) - ├─ Standalone instance - ├─ Data: In-memory only (no persistence configured) - └─ Network: docker-dataplane - -nextcloud (Nextcloud Stable) - ├─ Config: /home/jpmschweitzer/docker-data/nextcloud/config - ├─ Data: /mnt/media/nextcloud/data - └─ Dependencies: nextcloud-db, nextcloud-redis -``` - -### Target Setup -```yaml -postgres-shared (PostgreSQL 16) - ├─ New database: nextcloud - ├─ New user: nextcloud_user - └─ Database allocation: DB 3 - -redis-shared (Redis Alpine) - ├─ Database allocation: DB 3 (Nextcloud) - ├─ Existing DB 0: General cache - ├─ Existing DB 1: Authentik - └─ Existing DB 2: Gitea -``` - ---- - -## Migration Challenges - -### Critical Issue: MariaDB → PostgreSQL -⚠️ **Nextcloud cannot simply switch database types!** - -Nextcloud's database schema is different between MariaDB and PostgreSQL: -- Different data types (e.g., LONGTEXT vs TEXT) -- Different auto-increment handling -- Different JSON field types -- Different index structures - -**Options:** - -### Option A: Fresh Install + Data Migration (RECOMMENDED) -✅ **Pros:** -- Clean database schema -- Opportunity to optimize -- Lower risk of corruption -- Can test before switching - -❌ **Cons:** -- Must recreate users/settings -- Requires careful data migration -- More complex process - -### Option B: Database Conversion -✅ **Pros:** -- Preserves all settings -- Preserves user data - -❌ **Cons:** -- Complex conversion process -- High risk of data loss -- Nextcloud doesn't officially support this -- May leave corrupted data - -**RECOMMENDATION: Option A (Fresh Install)** - ---- - -## Fresh Installation Plan - -### Phase 1: Complete Cleanup - PURGE ALL DATA - -**Estimated Time:** 2 minutes - -⚠️ **DESTRUCTIVE OPERATION - REQUIRES EXPLICIT APPROVAL** ⚠️ - -The following will be PERMANENTLY DELETED: -- All Nextcloud containers (nextcloud, nextcloud-db, nextcloud-redis) -- All Nextcloud configuration (/home/jpmschweitzer/docker-data/nextcloud) -- All Nextcloud user files (/mnt/media/nextcloud) -- All Nextcloud database data - -**APPROVAL REQUIRED BEFORE EACH DELETION STEP** - -```bash -# Step 1: Stop and remove containers -# APPROVAL: Stop containers? (y/n) -docker stop nextcloud nextcloud-db nextcloud-redis 2>/dev/null || true -docker rm nextcloud nextcloud-db nextcloud-redis 2>/dev/null || true - -# Step 2: Delete config directory -# APPROVAL: Delete /home/jpmschweitzer/docker-data/nextcloud? (y/n) -sudo rm -rf /home/jpmschweitzer/docker-data/nextcloud - -# Step 3: Delete user data directory -# APPROVAL: Delete /mnt/media/nextcloud? (y/n) -sudo rm -rf /mnt/media/nextcloud - -# Step 4: Verify complete removal -ls /home/jpmschweitzer/docker-data/ | grep nextcloud # Should be empty -ls /mnt/media/ | grep nextcloud # Should be empty -``` - -### Phase 2: Prepare Shared Infrastructure - -**Estimated Time:** 5 minutes - -```bash -# 1. Create Nextcloud database in postgres-shared -docker exec -i postgres-shared psql -U postgres <<'EOF' --- Nextcloud database -CREATE DATABASE nextcloud; -CREATE USER nextcloud_user WITH PASSWORD 'GENERATE_NEW_PASSWORD_HERE'; -GRANT ALL PRIVILEGES ON DATABASE nextcloud TO nextcloud_user; -\c nextcloud -GRANT ALL ON SCHEMA public TO nextcloud_user; -ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO nextcloud_user; -ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO nextcloud_user; -EOF - -# 2. Update redis-shared documentation (already supports DB 3) -# No action needed - redis-shared already configured for multi-database -``` - -### Phase 3: Create Fresh Nextcloud Stack Configuration - -**Estimated Time:** 5 minutes - -Create new `nextcloud-shared.yml`: - -```yaml -services: - nextcloud: - image: nextcloud:stable - container_name: nextcloud - restart: unless-stopped - ports: - - "8082:80" - volumes: - # Fresh config directory - - /home/jpmschweitzer/docker-data/nextcloud/config:/var/www/html/config - # Fresh user data directory - - /mnt/media/nextcloud/data:/var/www/html/data - environment: - # PostgreSQL configuration - - POSTGRES_HOST=postgres-shared - - POSTGRES_DB=nextcloud - - POSTGRES_USER=nextcloud_user - - POSTGRES_PASSWORD=${NEXTCLOUD_DB_PASSWORD} - - # Redis configuration (Database 3) - - REDIS_HOST=redis-shared - - REDIS_HOST_PORT=6379 - - REDIS_DB_INDEX=3 - - # Timezone - - TZ=Europe/Amsterdam - depends_on: - - postgres-shared - - redis-shared - networks: - - docker-dataplane - deploy: - resources: - limits: - memory: 1G - -networks: - docker-dataplane: - external: true - name: docker-dataplane -``` - -### Phase 5: Deploy Fresh Nextcloud - -**Estimated Time:** 5 minutes - -```bash -# 1. Create new config directory -mkdir -p /home/jpmschweitzer/docker-data/nextcloud-shared/config - -# 2. Create .env file with database password -cat > /mnt/media/Projects/portainer-core/stacks/.env.nextcloud-shared < -EOF - -# 3. Deploy new stack -cd /mnt/media/Projects/portainer-core/stacks -docker compose -f nextcloud-shared.yml --env-file .env.nextcloud-shared up -d - -# 4. Wait for initialization -docker logs -f nextcloud -``` - -### Phase 6: Initial Setup & Configuration - -**Estimated Time:** 10 minutes - -```bash -# 1. Access Nextcloud web interface -# Navigate to: http://localhost:8082 or https://cloud.schweitz.net - -# 2. First-time setup wizard: -# - Admin username: admin -# - Admin password: -# - Data folder: /var/www/html/data (default) -# - Database: PostgreSQL -# - Database user: nextcloud_user -# - Database password: -# - Database name: nextcloud -# - Database host: postgres-shared - -# 3. Wait for installation (2-3 minutes) - -# 4. Configure trusted domains -docker exec -u www-data nextcloud php occ config:system:set trusted_domains 1 --value=cloud.schweitz.net -docker exec -u www-data nextcloud php occ config:system:set trusted_domains 2 --value=192.168.86.149 - -# 5. Configure Redis caching -docker exec -u www-data nextcloud php occ config:system:set redis host --value=redis-shared -docker exec -u www-data nextcloud php occ config:system:set redis port --value=6379 -docker exec -u www-data nextcloud php occ config:system:set redis dbindex --value=3 -docker exec -u www-data nextcloud php occ config:system:set memcache.local --value='\\OC\\Memcache\\APCu' -docker exec -u www-data nextcloud php occ config:system:set memcache.distributed --value='\\OC\\Memcache\\Redis' -docker exec -u www-data nextcloud php occ config:system:set memcache.locking --value='\\OC\\Memcache\\Redis' - -# 6. Optimize database -docker exec -u www-data nextcloud php occ db:add-missing-indices -docker exec -u www-data nextcloud php occ db:convert-filecache-bigint - -# 7. Configure background jobs -docker exec -u www-data nextcloud php occ background:cron -``` - -### Phase 7: Verify Fresh Installation - -**Estimated Time:** 5 minutes - -```bash -# 1. Verify admin user can login via web interface -# Navigate to: https://cloud.schweitz.net - -# 2. Check PostgreSQL connection -docker exec postgres-shared psql -U nextcloud_user -d nextcloud -c '\dt' - -# 3. Check Redis caching -docker exec redis-shared redis-cli -n 3 DBSIZE - -# 4. Verify storage location -docker exec -u www-data nextcloud php occ config:system:get datadirectory - -# 5. Test file upload/download -# Upload a test file via web interface -# Download it back -# Delete it -``` - -### Phase 8: Final Cleanup & Documentation - -**Estimated Time:** 2 minutes - -```bash -# 1. Update postgres-shared.yml documentation -# Add Nextcloud to "Applications Using This Database" list - -# 2. Update redis-shared.yml documentation -# Add "DB 3: Nextcloud (file locking, distributed cache)" - -# 3. Rename stack file -cd /mnt/media/Projects/portainer-core/stacks -mv nextcloud.yml nextcloud-mariadb-archived.yml -mv nextcloud-shared.yml nextcloud.yml - -# 4. Delete old archived stack (already purged data in Phase 1) -# All old containers and data already removed -``` - ---- - -## Rollback Plan - -⚠️ **NO ROLLBACK POSSIBLE** ⚠️ - -Since all old data is purged in Phase 1, there is no rollback option. - -If fresh installation fails: -1. Review error logs -2. Fix configuration issues -3. Retry fresh installation - -This is acceptable since Nextcloud is not in production use. - ---- - -## Testing Checklist - -After fresh installation, verify: - -- [ ] Admin login works -- [ ] File upload works -- [ ] File download works -- [ ] File delete works -- [ ] Redis caching active (`docker exec redis-shared redis-cli -n 3 DBSIZE` shows keys) -- [ ] PostgreSQL connection stable (`docker exec postgres-shared psql -U nextcloud_user -d nextcloud -c '\dt'` shows tables) -- [ ] Memory usage acceptable (<1GB for Nextcloud container) -- [ ] Nextcloud accessible via https://cloud.schweitz.net -- [ ] No errors in logs (`docker logs nextcloud`) - ---- - -## Resource Savings - -**Before Migration:** -- nextcloud-db (MariaDB): ~117 MB RAM -- nextcloud-redis: ~10 MB RAM -- **Total:** ~127 MB RAM + 2 containers - -**After Migration:** -- Shared postgres-shared: Already running (minimal additional overhead for one more DB) -- Shared redis-shared: Already running (DB 3 uses ~5-10 MB additional) -- **Savings:** ~110-120 MB RAM + 2 fewer containers to manage - -**Benefits:** -- Simplified infrastructure -- Centralized backups -- Better resource utilization -- Easier monitoring -- Consistent database management - ---- - -## Risks & Mitigation - -| Risk | Impact | Mitigation | -|------|--------|------------| -| Data loss during migration | HIGH | Full backups before starting, test on copy first | -| Incompatible plugins/apps | MEDIUM | Fresh install allows clean app selection | -| User resistance to re-setup | LOW | Minimal - same interface, same files | -| Extended downtime | MEDIUM | Plan migration during low-usage window | -| Redis DB conflict | LOW | Using dedicated DB 3, isolated from other apps | - ---- - -## Timeline - -**Total estimated time:** 20-30 minutes - -- Phase 1: Purge all data: 2 min -- Phase 2: Prepare PostgreSQL/Redis: 5 min -- Phase 3: Create stack config: 2 min -- Phase 4: Create directories: 1 min -- Phase 5: Deploy Nextcloud: 3 min -- Phase 6: Initial setup & config: 10 min -- Phase 7: Testing: 5 min -- Phase 8: Documentation: 2 min - -**Can be done anytime** - No production impact, no backups needed - ---- - -## Approval Required - -- [ ] Backup strategy approved -- [ ] Fresh install approach approved -- [ ] Downtime window approved -- [ ] Testing checklist reviewed -- [ ] Rollback plan understood -- [ ] Ready to proceed - ---- - -## Notes - -- **COMPLETE FRESH START** - All old data deleted -- Clean database, optimal performance from day one -- PostgreSQL generally faster than MariaDB for Nextcloud workloads -- Redis DB 3 dedicated to Nextcloud (isolated from other apps) -- No migration complexity - just a clean installation -- Ready for production use immediately after setup diff --git a/stacks/nextcloud.yml b/stacks/nextcloud.yml index 97eb6dc..e766e65 100644 --- a/stacks/nextcloud.yml +++ b/stacks/nextcloud.yml @@ -1,38 +1,4 @@ -version: '3.8' - -# Nextcloud - Cloud Storage with Database and Redis -# Backlog: Application Deployment -# Ports: 8082 -# GPU: No -# Storage: SSD (config/database), HDD (user data) - services: - nextcloud-db: - image: mariadb:10.11 - container_name: nextcloud-db - command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW - restart: unless-stopped - volumes: - # Database on SSD for performance - - /home/jpmschweitzer/docker-data/nextcloud/db:/var/lib/mysql - environment: - - MYSQL_ROOT_PASSWORD=xDgobrmzXOl+GgvBdXC9+z5v0OrWb29t - - MYSQL_PASSWORD=maF91Sw9is6Zb57JVxU/gPGP8O/DsxFq - - MYSQL_DATABASE=nextcloud - - MYSQL_USER=nextcloud - - TZ=Europe/Amsterdam - networks: - - docker-dataplane - - nextcloud-redis: - image: redis:alpine - container_name: nextcloud-redis - restart: unless-stopped - environment: - - TZ=Europe/Amsterdam - networks: - - docker-dataplane - nextcloud: image: nextcloud:stable container_name: nextcloud @@ -40,61 +6,106 @@ services: ports: - "8082:80" volumes: - # App config on SSD - - /home/jpmschweitzer/docker-data/nextcloud/config:/var/www/html - - # User data on HDD (large files) + # Fresh config directory + - /home/jpmschweitzer/docker-data/nextcloud/config:/var/www/html/config + # Fresh user data directory - /mnt/media/nextcloud/data:/var/www/html/data environment: - - MYSQL_HOST=nextcloud-db - - MYSQL_PASSWORD=maF91Sw9is6Zb57JVxU/gPGP8O/DsxFq - - MYSQL_DATABASE=nextcloud - - MYSQL_USER=nextcloud - - REDIS_HOST=nextcloud-redis + # PostgreSQL configuration + - POSTGRES_HOST=postgres-shared + - POSTGRES_DB=nextcloud + - POSTGRES_USER=nextcloud_user + - POSTGRES_PASSWORD=${NEXTCLOUD_DB_PASSWORD} + + # Redis configuration (Database 3) + - REDIS_HOST=redis-shared + - REDIS_HOST_PORT=6379 + - REDIS_DB_INDEX=3 + + # Timezone - TZ=Europe/Amsterdam - depends_on: - - nextcloud-db - - nextcloud-redis networks: - docker-dataplane + deploy: + resources: + limits: + memory: 1G networks: docker-dataplane: external: true name: docker-dataplane -# ⚠️ SECURITY WARNING: -# Change MYSQL_ROOT_PASSWORD and MYSQL_PASSWORD before deploying! -# Use strong, unique passwords. +# Nextcloud - Personal Cloud Storage (Using Shared Infrastructure) +# Port: 8082 +# GPU: No +# Dependencies: postgres-shared, redis-shared +# +# Prerequisites: +# +# 1. Shared infrastructure must be running: +# docker ps | grep -E 'postgres-shared|redis-shared' +# +# 2. Database and user already created in postgres-shared: +# - Database: nextcloud +# - User: nextcloud_user +# - Redis DB: 3 +# +# 3. Create .env file with: +# NEXTCLOUD_DB_PASSWORD= +# +# 4. Deploy this stack: +# cd /mnt/media/Projects/portainer-core/stacks +# docker compose -f nextcloud-shared.yml --env-file .env.nextcloud-shared up -d # # After Deployment: -# 1. Access http://localhost:8082 -# 2. First-time setup: -# - Create admin account (strong password!) +# +# 1. Wait for initialization (2-3 minutes) +# +# 2. Access web interface: http://localhost:8082 or https://cloud.schweitz.net +# +# 3. First-time setup wizard: +# - Admin username: admin +# - Admin password: # - Data folder: /var/www/html/data (default) -# - Database: MySQL/MariaDB -# - Database user: nextcloud -# - Database password: (the one you set above) +# - Database: PostgreSQL +# - Database user: nextcloud_user +# - Database password: # - Database name: nextcloud -# - Database host: nextcloud-db -# 3. Wait for installation (may take a few minutes) +# - Database host: postgres-shared +# # 4. Configure trusted domains: -# docker exec -u www-data nextcloud php occ config:system:set trusted_domains 1 --value=tower-of-joy -# docker exec -u www-data nextcloud php occ config:system:set trusted_domains 2 --value=192.168.x.x +# docker exec -u www-data nextcloud php occ config:system:set trusted_domains 1 --value=cloud.schweitz.net +# docker exec -u www-data nextcloud php occ config:system:set trusted_domains 2 --value=192.168.86.149 # -# Optimization (recommended): -# docker exec -u www-data nextcloud php occ db:add-missing-indices -# docker exec -u www-data nextcloud php occ db:convert-filecache-bigint -# docker exec -u www-data nextcloud php occ background:cron +# 5. Configure Redis caching: +# docker exec -u www-data nextcloud php occ config:system:set redis host --value=redis-shared +# docker exec -u www-data nextcloud php occ config:system:set redis port --value=6379 +# docker exec -u www-data nextcloud php occ config:system:set redis dbindex --value=3 +# docker exec -u www-data nextcloud php occ config:system:set memcache.local --value='\\OC\\Memcache\\APCu' +# docker exec -u www-data nextcloud php occ config:system:set memcache.distributed --value='\\OC\\Memcache\\Redis' +# docker exec -u www-data nextcloud php occ config:system:set memcache.locking --value='\\OC\\Memcache\\Redis' # -# Add cron job for background tasks: -# echo "*/5 * * * * docker exec -u www-data nextcloud php cron.php" | sudo tee -a /etc/crontab +# 6. Optimize database: +# docker exec -u www-data nextcloud php occ db:add-missing-indices +# docker exec -u www-data nextcloud php occ db:convert-filecache-bigint # -# Features: -# - File sync and share -# - Calendar and contacts -# - Collaborative editing -# - Photo gallery -# - Mobile apps (iOS/Android) -# - Desktop sync client -# - External storage support +# 7. Configure background jobs: +# docker exec -u www-data nextcloud php occ background:cron +# +# Connection Details: +# +# Database: +# - Host: postgres-shared (from containers) / localhost (from host) +# - Port: 5432 +# - Database: nextcloud +# - User: nextcloud_user +# +# Cache: +# - Host: redis-shared (from containers) / localhost (from host) +# - Port: 6379 +# - Database: 3 +# +# Resource Usage: +# - Nextcloud: 1GB RAM limit +# - Savings: ~110-120 MB RAM + 2 fewer containers (MariaDB + Redis removed) diff --git a/stacks/postgres-shared.yml b/stacks/postgres-shared.yml index 1991a32..e5d7ad2 100644 --- a/stacks/postgres-shared.yml +++ b/stacks/postgres-shared.yml @@ -132,4 +132,5 @@ networks: # Applications Using This Database: # - Authentik (identity provider) # - Gitea (git hosting) - migrated from dedicated instance +# - Nextcloud (personal cloud storage) - migrated from MariaDB # - Future applications as needed diff --git a/stacks/redis-shared.yml b/stacks/redis-shared.yml index dba7143..c0ccb21 100644 --- a/stacks/redis-shared.yml +++ b/stacks/redis-shared.yml @@ -66,7 +66,7 @@ networks: # DB 0: General cache (default, shared lightweight caching) # DB 1: Authentik (sessions, cache, message queue) # DB 2: Gitea (cache, sessions) -# DB 3: Open WebUI (cache, if needed) +# DB 3: Nextcloud (file locking, distributed cache, sessions) # DB 4-15: Reserved for future applications # # Connection Examples: @@ -144,6 +144,7 @@ networks: # Applications Using This Cache: # - Authentik (sessions, policies, background tasks) # - Gitea (sessions, cache, queues) - if migrated +# - Nextcloud (file locking, distributed cache, sessions) # - Future applications as needed # # Performance Tips: