security rework and memory optimilizations.

2025-11-17 08:48:00 +01:00
parent 72a653f494
commit e8eb2e954c
55 changed files with 8301 additions and 245 deletions
@@ -35,6 +35,9 @@ This is the `tower-of-joy` project - a containerized home server infrastructure
 **External Access Rule:** All public/external services MUST route through Nginx Proxy Manager (NPM) for Let's Encrypt SSL management and unified logging. Never expose service ports directly to the internet (except NPM and Headscale).
 **Service Maintenance**
 - **Stack Management** the portainer (and docker), NPM and Uptime Kuma services are managed through the core-api service. To maintain settings and configurations of these systems, read the documentation at http://tower-of-joy:8083/docs and prefer to use the api functions over direct reads an edits.
 **Service Integration Policy:** A service deployment is INCOMPLETE until cross-service integrations are implemented. Every new service MUST be integrated with:
 - **Uptime Kuma:** Add health check monitor (use `scripts/setup-kuma-monitors.sh` as guide)
 - **Organizr:** Configure service in dashboard (Settings → Tab Editor, Homepage Items)
@@ -2,7 +2,7 @@
 > Documentation of all deployed containers in the tower-of-joy infrastructure
 >
-> Last Updated: 2025-11-12
+> Last Updated: 2025-11-16
 ---
@@ -28,6 +28,65 @@ Portainer provides the web-based container management interface for the entire s
 ---
 ### PostgreSQL Shared
 PostgreSQL Shared is a centralized PostgreSQL 17 database server providing isolated database instances for multiple applications across the infrastructure, including Authentik (SSO), Gitea (Git hosting), and future services requiring relational database storage. It implements a shared infrastructure pattern where each application gets its own database and user credentials while sharing the same PostgreSQL instance for resource efficiency. The service stores all database data on the HDD with automated backups scheduled to the backups directory, providing persistent storage with volume-based data retention across container updates.
 | Property | Value |
 |----------|-------|
 | **Image** | `postgres:17` |
 | **Container Name** | `postgres-shared` |
 | **Access URL** | N/A (internal database server) |
 | **External Access** | No (docker-dataplane network only) |
 | **Port Mapping** | 5432:5432 (PostgreSQL) |
 | **Network Mode** | Bridge (docker-dataplane) |
 | **Restart Policy** | `unless-stopped` |
 | **Volume Mounts** | `/home/jpmschweitzer/docker-data/postgres-shared/data:/var/lib/postgresql/data`, `/home/jpmschweitzer/docker-data/postgres-shared/backups:/backups` |
 | **Environment** | `POSTGRES_PASSWORD=<secure-password>`, `POSTGRES_DB=postgres`, `TZ=Europe/Amsterdam`, `PGDATA=/var/lib/postgresql/data/pgdata` |
 | **Resource Limits** | None |
 | **GPU Required** | No |
 | **Dependencies** | docker-dataplane network |
 | **Databases** | `authentik` (Authentik SSO), `gitea` (Git hosting), `postgres` (default/admin) |
 | **Database Users** | `authentik_user`, `gitea_user`, `postgres` (superuser) |
 | **Health Check** | `pg_isready -U postgres` (30s interval) |
 | **Backup Strategy** | `/backups` volume for pg_dump exports |
 **Initialization**: Databases and users are created manually after first deployment:
 ```bash
 docker exec -i postgres-shared psql -U postgres <<'EOF'
  CREATE DATABASE authentik;
  CREATE USER authentik_user WITH PASSWORD '<password>';
  GRANT ALL PRIVILEGES ON DATABASE authentik TO authentik_user;
 EOF
 ```
 ---
 ### Redis Shared
 Redis Shared is a centralized Redis 7 key-value store providing cache, session storage, and message queue capabilities for multiple applications, with logical database isolation (DB 0-15) allowing each service to maintain separate keyspaces within the same Redis instance. It implements a shared infrastructure pattern where applications like Authentik use DB 0 for sessions/cache while future services can use DB 1-15, eliminating the need for separate Redis containers per application. The service stores data on the HDD for persistence across restarts, with AOF (Append-Only File) enabled for durability and optional RDB snapshots for backup points.
 | Property | Value |
 |----------|-------|
 | **Image** | `redis:7-alpine` |
 | **Container Name** | `redis-shared` |
 | **Access URL** | N/A (internal cache server) |
 | **External Access** | No (docker-dataplane network only) |
 | **Port Mapping** | 6379:6379 (Redis) |
 | **Network Mode** | Bridge (docker-dataplane) |
 | **Restart Policy** | `unless-stopped` |
 | **Volume Mounts** | `/home/jpmschweitzer/docker-data/redis-shared/data:/data` |
 | **Command** | `redis-server --appendonly yes --dir /data` |
 | **Resource Limits** | None |
 | **GPU Required** | No |
 | **Dependencies** | docker-dataplane network |
 | **Database Allocation** | DB 0: Authentik, DB 1-15: Available for future services |
 | **Persistence** | AOF (Append-Only File) enabled for durability |
 | **Health Check** | `redis-cli ping` returns PONG (30s interval) |
 | **Connection String** | `redis://redis-shared:6379/0` (DB 0), `redis://redis-shared:6379/1` (DB 1), etc. |
 ---
 ### Nginx Proxy Manager
 Nginx Proxy Manager serves as the unified reverse proxy and SSL certificate manager, providing a web-based interface for routing HTTP/HTTPS traffic to backend services with automatic Let's Encrypt certificate provisioning. It consolidates access to all web services through a single entry point with path-based or subdomain routing, eliminating the need to remember individual service ports. The service handles SSL termination, proxy host configuration, and access list management through an intuitive dashboard.
@@ -167,28 +226,6 @@ Netdata provides comprehensive real-time system performance monitoring with per-
 ---
 ### Heimdall
 Heimdall serves as a unified application dashboard providing organized, icon-based links to all homelab services with customizable layouts, colors, and application tiles for quick access without memorizing ports or URLs. It offers a single landing page for the entire infrastructure with support for application-specific widgets showing service stats, enhanced items with live data feeds, and pinned shortcuts for frequently accessed services. The interface reduces navigation complexity by centralizing access to Portainer, monitoring tools, media servers, and productivity applications.
 | Property | Value |
 |----------|-------|
 | **Image** | `linuxserver/heimdall:latest` |
 | **Container Name** | `heimdall` |
 | **Access URL** | http://192.168.86.149:8888 |
 | **External Access** | LAN only (dashboard) |
 | **Port Mapping** | 8888:80 (HTTP), 8889:443 (HTTPS) |
 | **Network Mode** | Bridge |
 | **Restart Policy** | `unless-stopped` |
 | **Volume Mounts** | `~/docker-data/heimdall:/config` |
 | **Environment** | `PUID=1000`, `PGID=1000`, `TZ=America/New_York` |
 | **Resource Limits** | None |
 | **GPU Required** | No |
 | **Dependencies** | None |
 | **Configuration** | Web-based (stored in volume) |
 ---
 ### Organizr
 Organizr serves as a comprehensive unified dashboard that consolidates all homelab services into a single tabbed interface with integrated homepage widgets showing real-time statistics from Jellyfin streams, Netdata metrics, Uptime Kuma status checks, and download client activity. It provides customizable authentication per-tab with support for SSO integration, user management with group-based access control, and a mobile-responsive interface for managing the entire infrastructure from anywhere. The service acts as a central hub replacing the need for multiple bookmarks or remembering service ports, offering both quick-access tabs and homepage cards with live data feeds from connected services.
@@ -268,7 +305,7 @@ The maintenance container runs scheduled automation tasks including nightly Dock
 ### Open WebUI
-Open WebUI is a feature-rich, self-hosted web interface for interacting with large language models via Ollama, providing a ChatGPT-like experience with support for multiple models, conversation history, RAG (Retrieval-Augmented Generation), web search integration, and user authentication. It offers a modern chat interface with streaming responses, markdown rendering, code syntax highlighting, and conversation management, enabling seamless switching between different LLM models and maintaining persistent chat histories in a local database. The service integrates directly with the local Ollama instance for GPU-accelerated inference without cloud dependencies, supporting features like web search via DuckDuckGo, document upload for context, and multi-user access with authentication.
+Open WebUI is a feature-rich, self-hosted web interface for interacting with large language models via Ollama, providing a ChatGPT-like experience with support for multiple models, conversation history, RAG (Retrieval-Augmented Generation), web search integration, and user authentication. It offers a modern chat interface with streaming responses, markdown rendering, code syntax highlighting, and conversation management, enabling seamless switching between different LLM models and maintaining persistent chat histories in a local database. The service integrates directly with the local Ollama instance for GPU-accelerated inference without cloud dependencies, supporting features like web search via DuckDuckGo, document upload for context, and multi-user access with authentication. It is documented at: https://docs.openwebui.com/
 | Property | Value |
 |----------|-------|
@@ -311,9 +348,44 @@ Core API provides OpenAI-compatible HTTP functions for Open WebUI, extending LLM
 | **GPU Required** | No (proxies requests to Ollama which uses GPU) |
 | **Dependencies** | Ollama (model inference), Open WebUI (consumes this API), ai-dataplane network |
 | **Framework** | FastAPI 0.115.0, Uvicorn 0.32.0, Pydantic 2.10.4, httpx 0.28.1 |
-| **Key Features** | OpenAI-compatible API (`/v1/chat/completions`, `/v1/models`), Model aliasing (OpenAI → local models), Streaming & non-streaming responses, Web scraping (Trafilatura, BeautifulSoup), Hot-reload development, OpenAPI spec |
+| **Key Features** | OpenAI-compatible API (`/v1/chat/completions`, `/v1/models`), Model aliasing (OpenAI → local models), Streaming & non-streaming responses, Web scraping (Trafilatura, BeautifulSoup), Hot-reload development, OpenAPI spec, Infrastructure management (Portainer, Uptime Kuma) |
 | **AI Orchestrator** | **Phase 1 Complete** - OpenAI API wrapper with model routing. Phase 2+ will add memory systems, multi-agent workflows, and tool integration. |
 | **Health Check** | `GET /health` (30s interval) - checks API status and Ollama connectivity |
 | **Monitoring API** | Full CRUD for Uptime Kuma monitors via Socket.IO: `GET /infrastructure/monitors`, `POST /infrastructure/monitors`, `GET /infrastructure/monitors/{id}`, `PUT /infrastructure/monitors/{id}`, `DELETE /infrastructure/monitors/{id}` |
 **Creating Monitors via API**:
 ```bash
 # Create a TCP port monitor for Redis
 curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
  -H "Content-Type: application/json" \
  -d '{
    "type": "port",
    "name": "Redis Shared - Port Check",
    "hostname": "redis-shared",
    "port": 6379,
    "interval": 60,
    "retryInterval": 60,
    "maxretries": 3,
    "notificationIDList": [],
    "accepted_statuscodes": ["200-299"]
  }'
 # Create a PostgreSQL database monitor (note URL-encoded password)
 curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
  -H "Content-Type: application/json" \
  -d '{
    "type": "postgres",
    "name": "PostgreSQL Shared",
    "interval": 60,
    "retryInterval": 60,
    "maxretries": 3,
    "notificationIDList": [],
    "accepted_statuscodes": ["200-299"],
    "databaseConnectionString": "postgres://user:password@postgres-shared:5432/postgres"
  }'
 ```
 **Important**: When creating database monitors with passwords containing special characters (`/`, `=`, `+`, etc.), URL-encode them in the connection string (e.g., `/` → `%2F`, `=` → `%3D`).
 ---
@@ -426,6 +498,8 @@ Gitea is a lightweight, self-hosted Git service providing repository hosting, is
 | Service | LAN URL | Internet Access | Primary Function |
 |---------|---------|-----------------|------------------|
 | **Portainer** | http://192.168.86.149:8001 | No | Container management |
 | **PostgreSQL Shared** | postgres-shared:5432 | No (internal) | Shared database server |
 | **Redis Shared** | redis-shared:6379 | No (internal) | Shared cache/session store |
 | **NPM** | http://192.168.86.149:81 | Yes (admin) | Reverse proxy admin |
 | **Code-Server** | https://code.schweitz.net | Yes | Browser-based IDE |
 | **Ollama** | http://192.168.86.149:11434 | No | ML model API |
@@ -435,7 +509,7 @@ Gitea is a lightweight, self-hosted Git service providing repository hosting, is
 | **Heimdall** | http://192.168.86.149:8888 | No | Service dashboard |
 | **Organizr** | https://home.schweitz.net | Yes | Unified dashboard |
 | **Open WebUI** | http://192.168.86.149:82 | No | LLM chat interface |
-| **Core API** | http://192.168.86.149:8083 | No | API functions for Open WebUI |
+| **Core API** | http://192.168.86.149:8083 | No | API functions & infrastructure mgmt |
 | **Jellyfin** | https://media.schweitz.net | Yes | Media streaming |
 | **Nextcloud** | https://cloud.schweitz.net | Yes | Cloud storage & sync |
 | **Gitea** | https://git.schweitz.net | Yes | Git repository hosting |
@@ -461,6 +535,8 @@ Gitea is a lightweight, self-hosted Git service providing repository hosting, is
 | Service | Config Location (SSD) | Data Location (HDD) | Typical Size |
 |---------|----------------------|---------------------|--------------|
 | **Portainer** | Docker volume: `portainer_data` | N/A | ~100MB |
 | **PostgreSQL Shared** | N/A | `~/docker-data/postgres-shared/data/` | Data: 100MB-5GB (depends on databases), Backups: variable |
 | **Redis Shared** | N/A | `~/docker-data/redis-shared/data/` | ~10-100MB (AOF + RDB snapshots) |
 | **NPM** | `~/docker-data/nginx-proxy-manager/` | N/A | ~50MB |
 | **Code-Server** | `~/.config/code-server/`, `~/docker-data/code-server/` | N/A | Config: ~5MB, Extensions: ~50-200MB, User data: ~50MB |
 | **Ollama** | `~/docker-data/ollama/models/` | Alt: `/mnt/media/ollama/` | 2-15GB per model |
@@ -484,15 +560,27 @@ Gitea is a lightweight, self-hosted Git service providing repository hosting, is
 ### Network Architecture
 **As of 2025-11-15**, all services have been consolidated to the `docker-dataplane` bridge network for simplified service discovery and inter-container communication. This consolidation replaced 12+ legacy networks with a single unified network, enabling all services to communicate using container names as DNS hostnames.
 | Network Name | Containers | Purpose |
 |--------------|------------|---------|
-| **host** | Portainer, NPM, Jellyfin, Netdata | Direct host port access, system visibility |
+| **docker-dataplane** | Ollama, Open WebUI, Core API, Qdrant, Uptime Kuma, PostgreSQL Shared, Redis Shared, Headscale, Nextcloud, Gitea, Samba, Watchtower, Maintenance, Organizr, Netdata | Unified service mesh for all containerized applications |
-| **ai-dataplane** | Ollama, Open WebUI, Core API, Qdrant, Uptime Kuma | AI service cluster |
+| **host** | Portainer, NPM | Direct host port access for infrastructure management |
-| **stacks_default** | Uptime Kuma, Netdata, Organizr, Watchtower, Maintenance | General infrastructure |
+
-| **stacks_headscale-network** | Headscale, Uptime Kuma | VPN control plane |
+**Benefits of Consolidation**:
-| **nextcloud_nextcloud-network** | Nextcloud, nextcloud-db, nextcloud-redis, Uptime Kuma | Cloud storage stack |
+- **Service Discovery**: All services reachable via `http://container-name:port` (e.g., `http://postgres-shared:5432`)
-| **gitea_gitea-network** | Gitea, gitea-db, Uptime Kuma | Git service stack |
+- **Simplified Monitoring**: Uptime Kuma can monitor all services on docker-dataplane
-| **samba_default** | Samba, Uptime Kuma | File sharing |
+- **Shared Infrastructure**: postgres-shared and redis-shared accessible to all applications
 - **Network Cleanup**: Removed 7 obsolete networks (stacks_default, ai-dataplane, various stack-specific networks)
 **Container Name Resolution Examples**:
 ```bash
 # From any container on docker-dataplane
 curl http://uptime-kuma:3001      # Uptime Kuma API
 curl http://ollama:11434           # Ollama LLM API
 psql -h postgres-shared -U postgres  # PostgreSQL connection
 redis-cli -h redis-shared          # Redis connection
 ```
 ---
@@ -517,5 +605,5 @@ Gitea is a lightweight, self-hosted Git service providing repository hosting, is
 ---
-*Last Updated: 2025-11-13*
+*Last Updated: 2025-11-16*
 *System: tower-of-joy (tower-of-joy v0.5.0-optimization)*
@@ -26,24 +26,31 @@
 ```
 ┌─────────────────────────────────────────┐
 │  Infrastructure Layer                   │
-│  ├── Portainer (8080) - Container mgmt │
+│  ├── Portainer (8001) - Container mgmt │
-│  ├── NPM (8000) - Reverse proxy        │
+│  ├── PostgreSQL Shared (5432) - DB     │
 │  ├── Redis Shared (6379) - Cache       │
 │  ├── NPM (81) - Reverse proxy          │
 │  └── Ollama (11434) - ML models [GPU]  │
 ├─────────────────────────────────────────┤
 │  Networking Layer                       │
-│  └── Headscale (8085) - Secure mesh    │
+│  ├── docker-dataplane - Service mesh   │
 │  └── Headscale (8085) - VPN mesh       │
 ├─────────────────────────────────────────┤
 │  Monitoring Layer                       │
 │  ├── Uptime Kuma (3001) - Uptime       │
 │  ├── Netdata (19999) - Metrics         │
 │  └── Organizr (9999) - Dashboard       │
 ├─────────────────────────────────────────┤
 │  Optimization Layer                     │
 │  ├── Watchtower - Auto-updates         │
 │  └── Maintenance - Automated backups   │
 ├─────────────────────────────────────────┤
-│  Application Layer (Backlog)            │
+│  Application Layer                      │
 │  ├── Open WebUI (82) - LLM chat UI     │
 │  ├── Core API (8083) - Infra mgmt      │
 │  ├── Jellyfin (8096) - Media [GPU]     │
 │  ├── Nextcloud (8082) - Cloud storage  │
 │  ├── Gitea (3002) - Git hosting        │
 │  └── Samba (445) - File shares         │
 └─────────────────────────────────────────┘
 ```
@@ -78,12 +85,12 @@ tower-of-joy/
 ## Documentation
 - **[CONTAINERS.md](CONTAINERS.md)** - Complete container reference guide with specs and access details
 - **[docs/SHARED_INFRASTRUCTURE_ARCHITECTURE.md](docs/SHARED_INFRASTRUCTURE_ARCHITECTURE.md)** - PostgreSQL/Redis shared infrastructure design
 - **[AGENTS.md](AGENTS.md)** - Guidelines for AI coding agents (conventions, testing, commits)
 - **[STATUS.md](STATUS.md)** - Current implementation phase and progress
 - **[CHANGELOG.md](CHANGELOG.md)** - Version history and completed work
 - **[SYSTEM.md](SYSTEM.md)** - Detailed hardware and software specs
 - **[containers/research.md](containers/research.md)** - Platform research and comparison
 - **[containers/implementation-plan.md](containers/implementation-plan.md)** - Detailed deployment guide
 ## Development Setup
@@ -138,19 +145,20 @@ make deploy-apps         # Deploy applications (Jellyfin, Nextcloud, Samba)
 | Service | Port | Description |
 |---------|------|-------------|
-| **Nginx Proxy Manager** | 8000 | Unified web interface entry point |
+| **Portainer** | 8001 | Container management UI |
-| **Portainer** | 8080 | Container management UI |
+| **PostgreSQL Shared** | 5432 | Shared database server (internal) |
-| **AMP** | 8081 | Game server management (existing) |
+| **Redis Shared** | 6379 | Shared cache server (internal) |
 | **Nginx Proxy Manager** | 81 | Reverse proxy admin |
 | **Open WebUI** | 82 | LLM chat interface |
 | **Ollama** | 11434 | ML model API |
-| **Core API** | 8083 | OpenAPI functions for Open WebUI |
+| **Core API** | 8083 | Infrastructure management API |
 | **Code-Server** | 8084 | Browser-based IDE (localhost only) |
-| **Headscale** | 8085 | Tailscale control server |
+| **Headscale** | 8085 | VPN control server |
 | **Jellyfin** | 8096 | Media streaming |
 | **Nextcloud** | 8082 | Cloud storage |
 | **Uptime Kuma** | 3001 | Service monitoring |
 | **Gitea** | 3002 | Git repository hosting |
 | **Netdata** | 19999 | System monitoring |
 | **Heimdall** | 8888 | Application dashboard |
 | **Organizr** | 9999 | Unified dashboard |
 ## GPU Services
@@ -212,6 +220,6 @@ Personal infrastructure project - not licensed for reuse.
 ---
-**Version:** 0.1.0-planning
+**Version:** 0.5.0-optimization
-**Last Updated:** 2025-11-11
+**Last Updated:** 2025-11-16
 **System:** tower-of-joy
@@ -0,0 +1,373 @@
 # Authentik Forward Authentication Deployment Summary
 ## Overview
 Successfully deployed Authentik forward authentication to all NPM-managed domains. All protected sites now require SSO authentication via Authentik before access is granted.
 ## Deployment Date
 2025-11-16
 ## Architecture
 ### Components
 1. **Authentik Server** (`authentik-server`)
   - Port: 9000
   - Container: `authentik-server`
   - Purpose: Main authentication server
 2. **Authentik Proxy Outpost** (`authentik-proxy-outpost`)
   - Port: 9001 (HTTP), 9300 (Metrics)
   - Container: `authentik-proxy-outpost`
   - Purpose: Forward authentication endpoint for NPM
   - Network: host mode (to match NPM)
   - Image: `ghcr.io/goauthentik/proxy:latest`
 3. **Nginx Proxy Manager** (`nginx-proxy-manager`)
   - Ports: 80, 81, 443
   - Purpose: Reverse proxy with SSL termination
   - Forward auth endpoint: `http://192.168.86.149:9001/outpost.goauthentik.io/auth/nginx`
 ### Authentication Flow
 ```
 1. User → https://home.schweitz.net
 2. NPM → auth_request to http://192.168.86.149:9001/outpost.goauthentik.io/auth/nginx
 3. Outpost → validates session with Authentik Server (port 9000)
 4. If not authenticated:
   a. NPM → redirects to /outpost.goauthentik.io/start?rd=https://home.schweitz.net/
   b. Outpost → redirects to https://auth.schweitz.net/if/flow/...
   c. User → logs in via Google OAuth
   d. Authentik → sets session cookie
   e. Redirect → back to https://home.schweitz.net/
 5. If authenticated:
   a. Request passes through with user headers
 ```
 ## Protected Domains
 All NPM proxy hosts now require authentication:
 1. **192.168.86.149** - Direct IP access
 2. **amp.schweitz.net** - AMP Server
 3. **cloud.schweitz.net** - Nextcloud
 4. **code.schweitz.net** - VS Code Server
 5. **git.schweitz.net** - Gitea
 6. **home.schweitz.net** - Home Assistant/Dashboard
 7. **media.schweitz.net** - Media Server
 8. **tatlock.schweitz.net** - Tatlock Services
 9. **tower-of-joy** - Tower of Joy Services
 ### Excluded Domains
 - **auth.schweitz.net** - Authentik itself (cannot protect the auth provider)
 ## Configuration Details
 ### Authentik Proxy Provider
 - **Name**: `npm-forward-auth-provider`
 - **ID**: 2
 - **Mode**: `forward_single`
 - **External Host**: `https://auth.schweitz.net`
 - **Token Validity**: 480 minutes (8 hours)
 - **Session Duration**: 480 minutes (8 hours)
 - **Rolling Expiration**: Active (sessions extend on use)
 ### Authentik Outpost
 - **Name**: `npm-forward-auth-outpost`
 - **ID**: `5fbba1f4-e50f-4d3f-94e8-8d9f30bf7a51`
 - **Type**: `proxy`
 - **Provider**: `npm-forward-auth-provider` (ID: 2)
 - **Token**: `aVJXnn3I44NhTYRYxGtbP3kJLEKQrTEMwu9IPVrhzN3NZV4vakgztcEf2L44`
 - **Container**: `authentik-proxy-outpost`
 - **Listen**: `0.0.0.0:9001` (HTTP), `0.0.0.0:9300` (metrics)
 ### NPM Configuration
 Each protected proxy host has this advanced nginx configuration:
 ```nginx
 # Authentik Forward Authentication
 # Send authentication requests to Authentik
 auth_request /outpost.goauthentik.io/auth/nginx;
 # Preserve authentication cookies
 auth_request_set $auth_cookie $upstream_http_set_cookie;
 add_header Set-Cookie $auth_cookie;
 # Get user information from Authentik
 auth_request_set $authentik_username $upstream_http_x_authentik_username;
 auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
 auth_request_set $authentik_email $upstream_http_x_authentik_email;
 auth_request_set $authentik_name $upstream_http_x_authentik_name;
 auth_request_set $authentik_uid $upstream_http_x_authentik_uid;
 # Pass user info to backend
 proxy_set_header X-authentik-username $authentik_username;
 proxy_set_header X-authentik-groups $authentik_groups;
 proxy_set_header X-authentik-email $authentik_email;
 proxy_set_header X-authentik-name $authentik_name;
 proxy_set_header X-authentik-uid $authentik_uid;
 # On authentication failure, redirect to Authentik login
 error_page 401 = @authentik_proxy_signin;
 location @authentik_proxy_signin {
    internal;
    add_header Set-Cookie $auth_cookie;
    return 302 /outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
 }
 # Authentik authentication endpoint
 location /outpost.goauthentik.io {
    proxy_pass http://192.168.86.149:9001/outpost.goauthentik.io;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header Host $host;
 }
 ```
 ## Deployment Steps Taken
 ### 1. Created Proxy Provider
 ```bash
 docker exec core-api /venv/bin/python /app/setup_authentik_forward_auth.py
 ```
 - Created `npm-forward-auth-provider` proxy provider (ID: 2)
 - Created `NPM Forward Auth` application (slug: `npm-forward-auth`)
 - Updated embedded outpost to include provider
 ### 2. Enabled Forward Auth on NPM Hosts
 ```bash
 docker exec core-api /venv/bin/python /app/enable_npm_forward_auth.py
 ```
 - Added forward auth configuration to all 9 NPM proxy hosts
 - Initially pointed to `http://192.168.86.149:9000` (incorrect)
 ### 3. Created Proxy Outpost
 ```bash
 python3 -c "..." # Created outpost via Authentik API
 ```
 - Created `npm-forward-auth-outpost` (ID: `5fbba1f4-e50f-4d3f-94e8-8d9f30bf7a51`)
 - Authentik auto-created container via Docker service connection
 - Retrieved token from auto-created container environment variables
 ### 4. Deployed Proxy Outpost Container
 ```bash
 docker run -d \
  --name authentik-proxy-outpost \
  --restart unless-stopped \
  --network host \
  -e AUTHENTIK_HOST=http://192.168.86.149:9000 \
  -e AUTHENTIK_TOKEN=aVJXnn3I44NhTYRYxGtbP3kJLEKQrTEMwu9IPVrhzN3NZV4vakgztcEf2L44 \
  -e AUTHENTIK_LISTEN__HTTP=0.0.0.0:9001 \
  -e AUTHENTIK_LISTEN__METRICS=0.0.0.0:9300 \
  ghcr.io/goauthentik/proxy:latest
 ```
 ### 5. Updated NPM Hosts to Port 9001
 ```bash
 docker exec core-api /venv/bin/python /app/fix_port_to_9001.py
 ```
 - Updated all 9 hosts to point to `http://192.168.86.149:9001` instead of port 9000
 ## Verification
 ### Testing Forward Auth
 ```bash
 # Test ping endpoint
 curl http://192.168.86.149:9001/outpost.goauthentik.io/ping
 # Expected: 204 No Content
 # Test unauthenticated request
 curl -I http://home.schweitz.net
 # Expected: 302 redirect to /outpost.goauthentik.io/start?rd=...
 ```
 ### Logs
 ```bash
 # Outpost logs
 docker logs authentik-proxy-outpost
 # Authentik server logs
 docker logs authentik-server
 # NPM logs
 docker logs nginx-proxy-manager
 ```
 ## Token & Session Management
 ### Token Validity
 - **Initial validity**: 480 minutes (8 hours)
 - **Rolling expiration**: Token stays active on use
 - **Idle timeout**: 8 hours after last activity
 - **Explicit logout**: Token invalidated immediately at https://auth.schweitz.net/if/user/#/settings
 ### Session Behavior
 - **Single login** protects all domains under `*.schweitz.net`
 - **Cookie domain**: Shared across all sites (set by individual domain)
 - **Persistent**: Survives browser restarts until expiration/logout
 - **Secure**: HTTPS-only, HttpOnly flag set
 ## User Management
 ### Adding Users
 1. Go to https://auth.schweitz.net/if/admin/
 2. Navigate to **Directory** → **Users**
 3. Click **Create** → **Create and enroll user**
 4. Enter user details
 5. Send enrollment invite
 ### Current Access Control
 **All authenticated users** can access all protected sites. To restrict:
 1. Go to https://auth.schweitz.net/if/admin/#/core/applications
 2. Select **NPM Forward Auth** application
 3. Go to **Policy / Group / User Bindings**
 4. Add specific users or groups
 ## Troubleshooting
 ### Check Outpost Status
 ```bash
 docker ps | grep authentik-proxy-outpost
 docker logs authentik-proxy-outpost --tail 50
 ```
 ### Check Authentik Server
 ```bash
 docker logs authentik-server --tail 50 | grep -i outpost
 ```
 ### Test Endpoints
 ```bash
 # Ping (should return 204)
 curl -I http://192.168.86.149:9001/outpost.goauthentik.io/ping
 # Auth endpoint (needs proper headers from nginx)
 curl -I http://home.schweitz.net
 ```
 ### Common Issues
 1. **502 Bad Gateway**: Outpost not running or not reachable
   - Check: `docker ps | grep authentik-proxy-outpost`
   - Fix: Restart outpost container
 2. **Redirect Loop**: Misconfigured nginx or outpost
   - Check: NPM advanced config points to port 9001
   - Check: Outpost logs for errors
 3. **Session Expires Too Quickly**: Token validity too short
   - Check: Provider token_validity setting (should be 480 minutes)
 ## Automation Scripts
 ### Enable Forward Auth on New Sites
 ```python
 from src.clients.npm_client import get_npm_client
 npm = get_npm_client()
 proxy = await npm.create_proxy_host(
    domain_names=["newsite.schweitz.net"],
    forward_host="backend-container",
    forward_port=8080,
    ssl_enabled=True
 )
 await npm.enable_authentik_forward_auth(proxy["id"])
 ```
 ### Bulk Operations
 ```bash
 # Enable on all hosts
 docker exec core-api /venv/bin/python /app/enable_npm_forward_auth.py
 # Update to port 9001
 docker exec core-api /venv/bin/python /app/fix_port_to_9001.py
 ```
 ## Security Considerations
 1. ✓ **HTTPS Required**: All sites use SSL termination at NPM
 2. ✓ **Secure Cookies**: HttpOnly and Secure flags prevent XSS/MITM
 3. ✓ **Token Rotation**: Tokens extend on activity for security
 4. ✓ **Audit Logging**: Authentik logs all authentication events
 5. ⚠ **MFA**: Not yet enabled (future enhancement)
 ## Monitoring
 ### Health Checks
 ```bash
 # Authentik server
 curl http://192.168.86.149:9000/-/health/live/
 # Proxy outpost
 curl http://192.168.86.149:9001/outpost.goauthentik.io/ping
 # Metrics
 curl http://192.168.86.149:9300/metrics
 ```
 ### Authentik Admin Dashboard
 - **URL**: https://auth.schweitz.net/if/admin/
 - **Events**: System → Events (view login attempts, failures)
 - **Active Sessions**: System → Tokens
 - **Outpost Status**: System → Outposts
 ## Future Enhancements
 1. **Multi-Factor Authentication**: Enable TOTP/SMS for high-security sites
 2. **Per-Site Access Control**: Different user groups for different domains
 3. **Rate Limiting**: Prevent brute force attacks
 4. **IP Whitelisting**: Allow certain IPs without auth
 5. **API Key Support**: Service-to-service authentication
 6. **Integration with Core API**: Expose auth status via REST API
 ## Related Documentation
 - **Main Documentation**: `FORWARD_AUTH_CONFIGURATION.md`
 - **SSO Progress**: `SSO_IMPLEMENTATION_PROGRESS.md`
 - **Security Plan**: `security-implementation-plan.md`
 - **Authentik Docs**: https://docs.goauthentik.io/docs/providers/proxy/
 ## Notes
 - The embedded Authentik outpost (ID: `fbbefe20-b2e1-4706-8533-7bfece51989c`) is for Authentik's own web UI, not for external forward auth
 - Forward auth requires a separate proxy outpost container listening on its own port (9001)
 - The outpost automatically created by Authentik via Docker service connection had the wrong port configuration
 - We manually deployed the outpost with correct settings using `docker run`
 - NPM uses host network mode, so the outpost must also use host mode or be accessible via host IP:port
 - Token keys are only visible once during creation in the Authentik UI; we retrieved it from the auto-created container's environment variables
 ## Completion Status
 ✅ **COMPLETE** - All protected sites now require Authentik authentication
 **Deployment successful as of 2025-11-16 15:41 UTC**
@@ -0,0 +1,259 @@
 # Authentik Forward Authentication Configuration
 ## Overview
 All NPM-managed domains now use Authentik for centralized Single Sign-On (SSO) authentication. When users access any protected site, they are automatically redirected to Authentik for login, then returned to their original destination.
 ## Authentication Flow
 1. **User accesses protected site** (e.g., https://portainer.schweitz.net)
 2. **NPM checks authentication** via Authentik forward auth
 3. **If not authenticated**: Redirect to https://auth.schweitz.net/outpost.goauthentik.io/start
 4. **User logs in** with Google (via Authentik)
 5. **Authentik sets session cookie** (8-hour validity)
 6. **Redirect back** to originally requested URL
 7. **Subsequent requests**: Automatically authenticated (no login required)
 ## Token & Session Configuration
 ### Token Validity
 - **Initial validity**: 8 hours (480 minutes)
 - **Activity-based extension**: Token stays active on use (rolling expiration)
 - **Idle timeout**: 8 hours after last activity
 - **Explicit logout**: Token invalidated immediately
 ### Session Behavior
 - **Single login** protects all domains under `*.schweitz.net`
 - **Cookie domain**: Shared across all sites
 - **Persistent**: Survives browser restarts (until expiration/logout)
 - **Secure**: HTTPS-only, HttpOnly flag set
 ## Protected Domains
 The following domains are protected with Authentik forward auth:
 1. **192.168.86.149** - Direct IP access
 2. **amp.schweitz.net** - AMP Server
 3. **cloud.schweitz.net** - Nextcloud
 4. **code.schweitz.net** - VS Code Server
 5. **git.schweitz.net** - Gitea
 6. **home.schweitz.net** - Home Assistant/Dashboard
 7. **media.schweitz.net** - Media Server
 8. **tatlock.schweitz.net** - Tatlock Services
 9. **tower-of-joy** - Tower of Joy Services
 ### Unprotected Domains
 - **auth.schweitz.net** - Authentik itself (cannot protect the auth provider)
 ## User Management
 ### Adding Users
 1. Go to https://auth.schweitz.net/if/admin/
 2. Navigate to **Directory** → **Users**
 3. Click **Create** → **Create and enroll user**
 4. Enter user details
 5. Send enrollment invite (they'll set up Google OAuth)
 ### User Access Control
 Currently, **all authenticated users** can access protected sites. To restrict access:
 1. Go to https://auth.schweitz.net/if/admin/#/core/applications
 2. Select **NPM Forward Auth** application
 3. Go to **Policy / Group / User Bindings**
 4. Add specific users or groups
 ### Group-Based Access (Future)
 You can create groups and assign different access levels:
 - `admin` - Full access to all sites
 - `family` - Access to media, home
 - `developers` - Access to code, git
 ## Logout
 Users can log out at: https://auth.schweitz.net/if/user/#/settings
 Click **Sign Out** to invalidate the session across all protected sites.
 ## Technical Implementation
 ### Authentik Components
 1. **Proxy Provider** (`npm-forward-auth-provider`)
   - Mode: `forward_single`
   - External Host: `https://auth.schweitz.net`
   - Token Validity: 480 minutes
 2. **Application** (`NPM Forward Auth`)
   - Links provider to user interface
   - Accessible at: https://auth.schweitz.net
 3. **Outpost** (`authentik Embedded Outpost`)
   - Handles authentication requests from NPM
   - Endpoint: `http://authentik-server:9000/outpost.goauthentik.io`
 ### NPM Configuration
 Each proxy host has advanced nginx configuration:
 ```nginx
 # Forward auth to Authentik
 auth_request /outpost.goauthentik.io/auth/nginx;
 # Preserve cookies
 auth_request_set $auth_cookie $upstream_http_set_cookie;
 add_header Set-Cookie $auth_cookie;
 # Extract user info
 auth_request_set $authentik_username $upstream_http_x_authentik_username;
 auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
 auth_request_set $authentik_email $upstream_http_x_authentik_email;
 # Pass to backend
 proxy_set_header X-authentik-username $authentik_username;
 proxy_set_header X-authentik-email $authentik_email;
 # Redirect on auth failure
 error_page 401 = @authentik_proxy_signin;
 location @authentik_proxy_signin {
    return 302 /outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
 }
 # Auth endpoint
 location /outpost.goauthentik.io {
    proxy_pass http://authentik-server:9000/outpost.goauthentik.io;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_pass_request_body off;
 }
 ```
 ## Backend Application Integration
 Protected applications receive user information via headers:
 - `X-authentik-username` - Username
 - `X-authentik-email` - Email address
 - `X-authentik-groups` - Comma-separated groups
 - `X-authentik-name` - Full name
 - `X-authentik-uid` - Unique user ID
 Applications can use these headers for:
 - Displaying user info
 - Audit logging
 - Role-based access control
 - Personalization
 ## Troubleshooting
 ### User Can't Log In
 1. **Check Authentik status**: `docker ps | grep authentik`
 2. **Check Authentik logs**: `docker logs authentik-server`
 3. **Verify Google OAuth**:
   - Go to https://auth.schweitz.net/if/admin/#/core/sources
   - Ensure Google source is enabled
 4. **Check user exists**:
   - Go to https://auth.schweitz.net/if/admin/#/identity/users
   - Verify user account is active
 ### Redirect Loop
 If users get stuck in a redirect loop:
 1. **Clear browser cookies** for `*.schweitz.net`
 2. **Check NPM config**: Ensure `/outpost.goauthentik.io` location exists
 3. **Restart NPM**: `docker restart npm`
 4. **Check outpost**: Verify provider is assigned to outpost
 ### 502 Bad Gateway
 If auth requests fail:
 1. **Check Authentik container**: `docker ps | grep authentik`
 2. **Verify network**: Ensure NPM can reach `authentik-server:9000`
 3. **Check NPM logs**: `docker logs npm`
 ### Session Expires Too Quickly
 If users are logged out unexpectedly:
 1. **Check token validity**: Should be 480 minutes (8 hours)
 2. **Verify rolling expiration**: Active use should extend session
 3. **Check system time**: Ensure Docker host time is correct
 ## Automation
 ### Adding Forward Auth to New Sites
 When creating new proxy hosts via core-api:
 ```python
 from src.clients.npm_client import get_npm_client
 npm = get_npm_client()
 # Create proxy host
 proxy = await npm.create_proxy_host(
    domain_names=["newsite.schweitz.net"],
    forward_host="backend-container",
    forward_port=8080,
    ssl_enabled=True
 )
 # Enable forward auth
 await npm.enable_authentik_forward_auth(proxy["id"])
 ```
 ### Bulk Enable/Disable
 To enable on all hosts:
 ```bash
 docker exec core-api /venv/bin/python /app/enable_npm_forward_auth.py
 ```
 ## Security Considerations
 1. **HTTPS Required**: Forward auth should only be used with HTTPS
 2. **Secure Cookies**: HttpOnly and Secure flags prevent XSS/MITM
 3. **Token Rotation**: Tokens are rotated on activity for security
 4. **Audit Logging**: Authentik logs all authentication events
 5. **MFA Support**: Can be enabled in Authentik for additional security
 ## Monitoring
 ### Check Authentication Status
 ```bash
 # Check Authentik health
 curl http://authentik-server:9000/-/health/live/
 # Check active sessions (in Authentik admin)
 # https://auth.schweitz.net/if/admin/#/events/log
 ```
 ### Monitor Failed Attempts
 Go to **System** → **Events** in Authentik admin to see:
 - Failed login attempts
 - Successful authentications
 - Token expirations
 - Policy violations
 ## Future Enhancements
 1. **Per-Site Access Control**: Different user groups for different domains
 2. **Multi-Factor Authentication**: SMS/TOTP for high-security sites
 3. **Rate Limiting**: Prevent brute force attacks
 4. **IP Whitelisting**: Allow certain IPs without auth
 5. **API Key Support**: Service-to-service authentication
 ## Related Documentation
 - **Security Implementation Plan**: `security-implementation-plan.md`
 - **SSO Progress**: `SSO_IMPLEMENTATION_PROGRESS.md`
 - **OIDC Configuration**: `OIDC_CONFIGURATION.md`
 - **Authentik Docs**: https://docs.goauthentik.io/docs/providers/proxy/
@@ -0,0 +1,167 @@
 # OIDC Authentication Configuration
 ## Overview
 Core-API has been configured with OIDC authentication support using Authentik as the identity provider. This provides secure authentication for infrastructure management endpoints.
 ## Current Status
 **OIDC is currently DISABLED** (`oidc_enabled=false` in config.py)
 This allows:
 - Internal services to access the API without authentication
 - Direct API access from the Docker network
 - Backward compatibility with existing integrations
 ## Authentik Configuration
 ### Provider Details
 - **Provider Name**: `core-api-provider`
 - **Provider ID**: `1`
 - **Client ID**: `core-api`
 - **Client Secret**: `WfsY0iIVOmO1wvXn1u8jwa08eiQsn2yVf9toBwVyEqps3M98nOZwJMgDOqH5PNZGM6wIxKTwlwYemtaOUg9u5bocd0EqShyoe4yhBQq4SDd0svyArILHZeGHFVEqUi4d`
 - **Issuer**: `https://auth.schweitz.net/application/o/core-api/`
 ### Application Details
 - **Application Name**: `Core API`
 - **Slug**: `core-api`
 - **Launch URL**: `http://localhost:8083/docs`
 ### Redirect URIs
 - `http://localhost:8083/docs/oauth2-redirect`
 - `https://core-api.schweitz.net/docs/oauth2-redirect`
 - `http://192.168.86.149:8083/docs/oauth2-redirect`
 ## Enabling OIDC Authentication
 When ready to enable OIDC authentication for external access:
 ### 1. Update core-api Configuration
 In `/home/jpmschweitzer/Projects/portainer-core/services/core-api/src/config.py`:
 ```python
 # OIDC Authentication (Authentik)
 oidc_enabled: bool = True  # Change from False to True
 oidc_issuer: str = "https://auth.schweitz.net/application/o/core-api/"
 oidc_audience: str = "core-api"
 oidc_client_secret: str = "WfsY0iIVOmO1wvXn1u8jwa08eiQsn2yVf9toBwVyEqps3M98nOZwJMgDOqH5PNZGM6wIxKTwlwYemtaOUg9u5bocd0EqShyoe4yhBQq4SDd0svyArILHZeGHFVEqUi4d"
 ```
 ### 2. Restart core-api
 ```bash
 docker restart core-api
 ```
 ### 3. Test Authentication
 1. Visit: http://localhost:8083/docs
 2. Click the "Authorize" button
 3. Log in with Google via Authentik
 4. Access protected endpoints
 ## Protected Endpoints
 When OIDC is enabled, the following endpoints require admin authentication:
 ### Infrastructure Management (Write Operations)
 - `POST /infrastructure/services` - Deploy stack
 - `PUT /infrastructure/services/{name}` - Update stack
 - `DELETE /infrastructure/services/{name}` - Delete stack
 - `POST /infrastructure/proxy` - Create proxy host
 - `POST /infrastructure/services/{name}/stop` - Stop service
 - `POST /infrastructure/services/{name}/start` - Start service
 ### Monitoring Management
 - `POST /infrastructure/monitors` - Create monitor
 - `PUT /infrastructure/monitors/{monitor_id}` - Update monitor
 - `DELETE /infrastructure/monitors/{monitor_id}` - Delete monitor
 ### Read Endpoints (Public)
 All GET endpoints remain publicly accessible:
 - `/health` - Health check
 - `/infrastructure/health` - Infrastructure health
 - `/infrastructure/services` - List services
 - `/infrastructure/ports` - List ports
 - `/infrastructure/domains` - List domains
 - `/infrastructure/monitors` - List monitors
 ## Internal Network Access
 **Important**: After enabling OIDC for external users, internal services still need unrestricted access.
 ### Future Enhancement Options
 1. **Network-based Authentication**
   - Check if request originates from `docker-dataplane` network
   - Allow requests from internal IPs without auth
   - Require OIDC only for external requests
 2. **Service Accounts**
   - Create machine tokens for internal services
   - Use Bearer token authentication for service-to-service calls
   - Keep OIDC for user authentication
 3. **NPM Proxy Layer**
   - External domain (https://core-api.schweitz.net) → Requires Authentik SSO
   - Internal access (http://core-api:8083) → No authentication
 ## Admin Groups
 Users must be members of one of these Authentik groups to access protected endpoints:
 - `admin`
 - `authentik Admins`
 Configure user group membership in Authentik admin panel:
 https://auth.schweitz.net/if/admin/#/identity/users
 ## Authentik Management
 - **Admin Panel**: https://auth.schweitz.net/if/admin/
 - **Applications**: https://auth.schweitz.net/if/admin/#/core/applications
 - **Providers**: https://auth.schweitz.net/if/admin/#/core/providers
 ## API Token Management
 The Authentik API token used for automation is stored in:
 `/home/jpmschweitzer/Projects/portainer-core/services/core-api/src/credentials.py`
 ```python
 AUTHENTIK_CORE_API_TOKEN = "peXM0EzDv2Wiwycbfm3cE5O44IMAOR8ntUwKVP977yFvVopCzDlKY8tymlMM"
 ```
 To create additional tokens: https://auth.schweitz.net/if/admin/#/identity/tokens
 ## Security Considerations
 1. **Client Secret**: Stored in config.py (gitignored), consider moving to environment variable
 2. **Token Validation**: Tokens are validated using JWKS from Authentik
 3. **Token Expiry**: Access tokens valid for 60 minutes, refresh tokens for 30 days
 4. **SSL**: All production endpoints should use HTTPS (via NPM)
 5. **Admin Access**: Restrict admin group membership carefully
 ## Troubleshooting
 ### Login Issues
 - Check Authentik service status: `docker ps | grep authentik`
 - Check Authentik logs: `docker logs authentik-server`
 - Verify redirect URIs match exactly in Authentik provider config
 ### Token Validation Errors
 - Verify OIDC issuer URL is correct
 - Check core-api logs: `docker logs core-api`
 - Ensure Authentik is accessible from core-api container
 ### Permission Denied (403)
 - Verify user is member of `admin` group in Authentik
 - Check user claims in JWT token at https://jwt.io
 ## Related Documentation
 - **Security Implementation Plan**: `/home/jpmschweitzer/Projects/portainer-core/security-implementation-plan.md`
 - **SSO Progress**: `/home/jpmschweitzer/Projects/portainer-core/SSO_IMPLEMENTATION_PROGRESS.md`
 - **Authentik Docs**: https://docs.goauthentik.io/
@@ -0,0 +1,292 @@
 # Shared Infrastructure Architecture
 **Purpose:** Centralized PostgreSQL and Redis services for all homelab stacks
 **Benefits:** Resource efficiency, easier maintenance, unified backups, centralized monitoring
 ---
 ## Architecture Overview
 ```
 ┌─────────────────────────────────────────────────────────────┐
 │                  Application Stacks                          │
 │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
 │  │Authentik │  │  Gitea   │  │ Future   │  │ Future   │   │
 │  │          │  │          │  │  Stack   │  │  Stack   │   │
 │  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘   │
 │       │             │              │              │          │
 └───────┼─────────────┼──────────────┼──────────────┼─────────┘
        │             │              │              │
        └─────────────┴──────────────┴──────────────┘
                      │
        ┌─────────────▼──────────────────────────────┐
        │      Unified Data Plane Network            │
        │          (docker-dataplane)                │
        └─────────────┬──────────────────────────────┘
                      │
        ┌─────────────┴──────────────┐
        │                            │
   ┌────▼──────┐            ┌────────▼────┐
   │PostgreSQL │            │    Redis    │
   │  Shared   │            │   Shared    │
   │           │            │             │
   │ Databases:│            │ DB 0: Cache │
   │  - auth   │            │ DB 1: Auth  │
   │  - gitea  │            │ DB 2: Gitea │
   │  - future │            │ DB 3-15: .. │
   └───────────┘            └─────────────┘
 ```
 ---
 ## Design Principles
 ### 1. **Database Isolation**
 - Each application gets its own PostgreSQL database within the shared instance
 - Each application gets its own Redis database number (0-15)
 - Separate credentials per application for security
 ### 2. **Network Architecture**
 - **Unified network:** `docker-dataplane` (external, bridge)
 - All application containers connect to this single network
 - Simplified connectivity: services discover each other by container name
 - Replaces per-stack networks (ai-dataplane, nextcloud-network, etc.)
 ### 3. **Resource Allocation**
 - PostgreSQL: No hard limits (homelab resource availability)
 - Redis: No hard limits (lightweight Alpine image)
 - Shared instances more efficient than per-stack deployments
 ### 4. **Backup Strategy**
 - Single PostgreSQL backup covers all databases
 - Automated pg_dumpall for disaster recovery
 - Redis persistence: AOF + RDB snapshots
 ### 5. **Security Model**
 - Each app has dedicated PostgreSQL user with access only to its database
 - Redis AUTH with per-database passwords (optional)
 - Network-level isolation via Docker networks
 ---
 ## Database Allocation Plan
 ### PostgreSQL Databases
 | Database Name | Application | User | Purpose |
 |---------------|-------------|------|---------|
 | `authentik` | Authentik | `authentik_user` | User/group/policy storage |
 | `gitea` | Gitea | `gitea_user` | Git repos, users, issues |
 | `future_app1` | TBD | `app1_user` | Reserved |
 | `future_app2` | TBD | `app2_user` | Reserved |
 **Note:** Existing services stay as-is:
 - Nextcloud: MariaDB (existing, not migrated)
 - Others can migrate over time if beneficial
 ### Redis Database Numbers
 | DB# | Application | Purpose |
 |-----|-------------|---------|
 | 0 | Authentik | Sessions, cache, message queue |
 | 1 | Available | Reserved for future applications |
 | 2 | Available | Reserved for future applications |
 | 3-15 | Available | Reserved for future applications |
 **Note:** Each application uses a dedicated DB number to prevent key collisions while sharing the same Redis instance.
 ---
 ## Connection Configuration
 ### PostgreSQL Connection Strings
 **From Docker containers:**
 ```
 Host: postgres-shared
 Port: 5432
 Database: authentik
 User: authentik_user
 Password: <app-specific-password>
 ```
 **From host:**
 ```
 Host: localhost
 Port: 5432
 Database: authentik
 User: authentik_user
 Password: <app-specific-password>
 ```
 ### Redis Connection Strings
 **From Docker containers:**
 ```
 redis://redis-shared:6379/0  (for Authentik, DB 0)
 redis://redis-shared:6379/1  (for future apps, DB 1)
 redis://redis-shared:6379/2  (for future apps, DB 2)
 ```
 **From host:**
 ```
 redis://localhost:6379/0
 ```
 ---
 ## Migration Strategy
 ### Phase 1: Deploy Shared Infrastructure ✅ **COMPLETE**
 1. ✅ Deployed `postgres-shared.yml` and `redis-shared.yml` via Portainer
 2. ✅ Verified PostgreSQL 17 and Redis 7 running on docker-dataplane
 3. ✅ Created initial databases and users (authentik, gitea)
 4. ✅ Both services monitored via Uptime Kuma
 ### Phase 2: New Services (Authentik) 🚧 **IN PROGRESS**
 1. ⏳ Deploy Authentik pointing to shared services
 2. ⏳ Test thoroughly
 3. ⏳ Validate no performance degradation
 ### Phase 3: Network Consolidation ✅ **COMPLETE**
 1. ✅ All services migrated to docker-dataplane network
 2. ✅ Removed 7 obsolete Docker networks
 3. ✅ 18 containers on unified network for service discovery
 ### Phase 4: Migrate Existing Services (Optional)
 1. **Gitea**: Already uses PostgreSQL
   - Export existing database
   - Create gitea database in shared PostgreSQL
   - Import data
   - Update Gitea stack to use shared PostgreSQL
   - Remove old gitea-db container
 2. **Other services**: Evaluate case-by-case
   - Nextcloud: Keep MariaDB (complex migration, low benefit)
   - Future services: Use shared from day 1
 ---
 ## Advantages
 ✅ **Resource Efficiency**
 - One PostgreSQL instance: ~1GB RAM vs ~300MB per instance
 - Saves ~700MB RAM per additional service using PostgreSQL
 ✅ **Operational Simplicity**
 - Single backup process for all PostgreSQL databases
 - Centralized monitoring and health checks
 - Easier version upgrades (upgrade once, affects all)
 ✅ **Performance**
 - Shared connection pooling
 - Better resource utilization
 - Optimized caching with shared Redis
 ✅ **Scalability**
 - Add new applications without deploying new database instances
 - Up to 15 Redis databases (more than enough for homelab)
 ---
 ## Disadvantages & Mitigations
 ⚠️ **Single Point of Failure**
 - **Mitigation:** Health checks, automated restarts, regular backups
 - **Acceptable for homelab:** VPN access ensures admin can fix issues
 ⚠️ **Resource Contention**
 - **Mitigation:** PostgreSQL connection limits per database
 - **Mitigation:** Redis max memory policy (LRU eviction)
 - **Monitoring:** Track per-database usage
 ⚠️ **Version Lock-In**
 - **Mitigation:** Use latest stable PostgreSQL version (17)
 - **Mitigation:** Test upgrades in staging before production deployment
 ---
 ## Monitoring & Maintenance
 ### Health Checks
 - PostgreSQL: `pg_isready` every 30s
 - Redis: `redis-cli ping` every 30s
 - Application connectivity tests
 ### Uptime Kuma Integration ✅ **DEPLOYED**
 Both shared services are monitored via Uptime Kuma with automatic monitor creation through the Core API:
 **PostgreSQL Monitor** (ID 20):
 ```bash
 curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
  -H "Content-Type: application/json" \
  -d '{
    "type": "postgres",
    "name": "PostgreSQL Shared",
    "interval": 60,
    "retryInterval": 60,
    "maxretries": 3,
    "notificationIDList": [],
    "accepted_statuscodes": ["200-299"],
    "databaseConnectionString": "postgres://postgres:<url-encoded-password>@postgres-shared:5432/postgres"
  }'
 ```
 **Redis Monitor** (ID 18):
 ```bash
 curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
  -H "Content-Type: application/json" \
  -d '{
    "type": "port",
    "name": "Redis Shared - Port Check",
    "hostname": "redis-shared",
    "port": 6379,
    "interval": 60,
    "retryInterval": 60,
    "maxretries": 3,
    "notificationIDList": [],
    "accepted_statuscodes": ["200-299"]
  }'
 ```
 **Note:** When monitoring PostgreSQL with passwords containing special characters, URL-encode them (`/` → `%2F`, `=` → `%3D`).
 ### Backup Schedule
 - **PostgreSQL:** Manual pg_dump to `/backups/` volume (automated backups pending)
 - **Redis:** AOF persistence (real-time) enabled via `--appendonly yes`
 ### Performance Monitoring
 - Query: `SELECT datname, numbackends FROM pg_stat_database;` (active connections)
 - Redis: `INFO stats` (keyspace usage per database)
 - Uptime Kuma dashboard: Real-time availability tracking
 ### Upgrade Path
 1. Backup all databases
 2. Test upgrade with docker-compose override
 3. Deploy new version
 4. Verify all applications connect successfully
 5. Rollback if issues detected
 ---
 ## Implementation Status
 1. ✅ Review architecture design
 2. ✅ Create `postgres-shared.yml` and `redis-shared.yml` stacks
 3. ✅ Deploy shared PostgreSQL 17 and Redis 7 via Portainer
 4. ✅ Create initial databases (authentik, gitea)
 5. ✅ Consolidate all services to docker-dataplane network
 6. ✅ Implement Uptime Kuma monitoring via Core API
 7. ✅ Document connection patterns and deployment procedures
 8. ⏳ Update `authentik.yml` to use shared services (pending)
 9. ⏳ Test Authentik with shared infrastructure (pending)
 ---
 ## Future Enhancements
 - **PostgreSQL Read Replicas** (if needed for heavy read workloads)
 - **Redis Sentinel** (high availability, probably overkill for homelab)
 - **PgBouncer** (connection pooling if >100 connections needed)
 - **Prometheus + Grafana** (metrics visualization)
@@ -0,0 +1,115 @@
 # SSO Implementation Progress Tracker
 **Project:** Google OAuth SSO for Homelab Infrastructure
 **Started:** 2025-11-15
 **Status:** 🚧 Phase 1 - In Progress
 ---
 ## Overview
 Implementing Single Sign-On (SSO) using:
 - **Identity Provider:** Authentik
 - **Authentication Source:** Google OAuth (Workspace + Gmail)
 - **In-Scope Services:** core-api, Nextcloud, Jellyfin, Gitea, Open WebUI, Organizr, code-server
 ---
 ## Phase 1: Foundation (Week 1)
 **Goal:** Deploy Authentik, configure Google OAuth, protect core-api
 ### Task Checklist
 - [x] **1.1 Deploy Authentik Stack** *(In Progress)*
  - [x] Create `stacks/authentik.yml`
  - [x] Generate secrets (PostgreSQL password, Authentik secret key)
  - [x] Create `.env.authentik` file
  - [x] Create data directories
  - [ ] Deploy via Portainer
  - [ ] Verify services running (4/4: server, worker, postgresql, redis)
  - [ ] Complete initial setup wizard
  - [ ] Access admin portal
 - [ ] **1.2 Configure NPM Proxy**
  - [ ] Create proxy host: `auth.schweitz.net` → `authentik-server:9000`
  - [ ] Enable SSL with Let's Encrypt
  - [ ] Test HTTPS access
  - [ ] Verify health endpoint
 - [ ] **1.3 Google OAuth Setup**
  - [ ] Create/configure Google Cloud Project
  - [ ] Set up OAuth consent screen
  - [ ] Create OAuth 2.0 credentials
  - [ ] Note Client ID and Client Secret
  - [ ] Configure authorized redirect URIs
 - [ ] **1.4 Configure Google Source in Authentik**
  - [ ] Add Google OAuth source
  - [ ] Configure scopes: openid, email, profile
  - [ ] Test login with Google Workspace account
  - [ ] Test login with Gmail account
  - [ ] Verify user profile synced
 - [ ] **1.5 Implement core-api OIDC Authentication**
  - [ ] Add dependencies: PyJWT, python-jose
  - [ ] Create `src/auth/oidc.py` module
  - [ ] Update `src/config.py` with OIDC settings
  - [ ] Create Authentik OIDC provider for core-api
  - [ ] Protect infrastructure endpoints
  - [ ] Update OpenAPI docs with security scheme
 - [ ] **1.6 Testing & Validation**
  - [ ] Test unauthenticated API request (expect 401)
  - [ ] Test authenticated API request with valid token
  - [ ] Verify user claims available in endpoints
  - [ ] Test token expiration handling
  - [ ] Test admin-only endpoints
  - [ ] Update widget for OAuth flow
 ---
 ## Progress Log
 ### 2025-11-15 - 21:20 CET
 **[Completed]** Shared Infrastructure Architecture
 - ✅ Designed shared PostgreSQL + Redis architecture
 - ✅ Created `SHARED_INFRASTRUCTURE_ARCHITECTURE.md` documentation
 - ✅ Created separate stacks for modularity:
  - `postgres-shared.yml` (centralized database)
  - `redis-shared.yml` (centralized cache)
  - `authentik-shared.yml` (using shared backends)
 - ✅ Generated secure credentials (all passwords 32-byte random)
 - ✅ Created unified `docker-dataplane` network
 - ✅ Created PostgreSQL init script for multi-database setup
 - ✅ Created environment files:
  - `.env.postgres` (PostgreSQL + app DB passwords)
  - `.env.authentik-shared` (Authentik config)
 **Architecture Benefits:**
 - Resource savings: ~400MB RAM per service using shared infrastructure
 - Centralized backups and monitoring
 - Easier maintenance and upgrades
 - Modular deployment (PostgreSQL and Redis as separate stacks)
 **[Next]** Deploy shared infrastructure, then Authentik
 ---
 ## Next: Create Authentik Stack
 Creating `stacks/authentik.yml` with:
 - authentik-server
 - authentik-worker
 - PostgreSQL database
 - Redis cache
 Expected resources: ~500MB RAM, 1.5 CPU, 5GB storage
 ---
 ## Notes
 - **Architecture Decision:** FastAPI native OIDC for core-api (not NPM forward auth)
 - **Out of Scope:** Infrastructure tools, data providers, local services
 - **Security:** All passwords/secrets via environment variables, not committed to git
@@ -0,0 +1,223 @@
 # Organizr Service Control Widget
 A beautiful, responsive widget for managing on-demand services from your Organizr dashboard.
 ## Features
 - ✨ **Real-time Status** - Live service status with container counts
 - 🎮 **One-Click Control** - Start/Stop services with a single click
 - 🔒 **Safety First** - Always-on services are protected and clearly marked
 - 🎨 **Beautiful UI** - Dark theme that matches Organizr
 - ⚡ **Auto-Refresh** - Updates every 10 seconds
 - 📱 **Responsive** - Works on desktop, tablet, and mobile
 ## Screenshots
 ### Service Cards
 Each service shows:
 - Service name
 - Running status (Running/Stopped with container counts)
 - Start/Stop buttons (disabled when not applicable)
 - "ALWAYS ON" badge for infrastructure services
 ## Installation
 ### Method 1: Organizr Custom Homepage Item (Recommended)
 1. **Copy the widget file** to a web-accessible location:
   ```bash
   # If you have a web server serving files from /var/www/html:
   sudo cp service-control.html /var/www/html/widgets/
   # Or use Organizr's public directory:
   cp service-control.html /path/to/organizr/plugins/widgets/
   ```
 2. **Add to Organizr Homepage**:
   - Open Organizr
   - Go to **Settings** → **Customize** → **Homepage Items**
   - Click **Add New Item**
   - Configure:
     - **Name**: "Service Control"
     - **Category**: Custom
     - **Type**: iFrame
     - **URL**: `http://localhost/widgets/service-control.html` (adjust path)
     - **Minimum Authentication**: User
     - **Enabled**: Yes
   - Save
 3. **Add to Homepage**:
   - Go to **Settings** → **Customize** → **Appearance**
   - Edit your homepage layout
   - Add the "Service Control" item to desired location
   - Save
 ### Method 2: Organizr Custom HTML Tab
 1. **Open Organizr Settings**:
   - Settings → **Tab Editor**
 2. **Add New Tab**:
   - Click **Add Tab**
   - Configure:
     - **Tab Name**: "Services"
     - **Tab URL**: Leave empty
     - **Category**: Custom
     - **Type**: iFrame
     - **Image**: `images/tabs/services.png` (or your choice)
 3. **Add Custom HTML**:
   - In the same tab configuration, find **Custom HTML** section
   - Copy and paste the entire contents of `service-control.html`
   - Save
 4. **Access the Tab**:
   - The "Services" tab will now appear in your Organizr sidebar
 ### Method 3: Nginx Reverse Proxy Integration
 If you want to serve the widget through Nginx Proxy Manager:
 1. **Create a location** in your Organizr proxy host:
   ```nginx
   location /widgets/ {
       alias /path/to/portainer-core/organizr-widgets/;
       autoindex off;
   }
   ```
 2. **Access via**: `https://your-organizr-domain.com/widgets/service-control.html`
 ## Configuration
 ### Changing API Endpoint
 If your core-api is not on `localhost:8083`, edit the widget file:
 ```javascript
 const API_BASE = 'http://your-server:8083';  // Change this line
 ```
 ### Adjusting Auto-Refresh Interval
 Default is 10 seconds. To change:
 ```javascript
 setInterval(fetchServices, 10000);  // Change 10000 to desired milliseconds
 ```
 ### Customizing Displayed Services
 By default, the widget shows all stoppable services (excludes always-on infrastructure).
 To filter specific services, modify the `renderServices()` function:
 ```javascript
 const stoppableServices = services.filter(s =>
    !isAlwaysOn(s.name) &&
    ['jellyfin', 'nextcloud', 'gitea', 'ai-stack'].includes(s.name)  // Add this line
 );
 ```
 ## Troubleshooting
 ### "Failed to connect to API"
 **Problem**: Widget shows red error message
 **Solutions**:
 1. Verify core-api is running: `docker ps | grep core-api`
 2. Check core-api URL is correct (localhost vs IP address)
 3. If accessing from remote, change `API_BASE` to full URL
 4. Check browser console for CORS errors
 ### CORS Issues
 If accessing widget from a different domain than core-api:
 **Option 1**: Update core-api CORS settings in `src/config.py`:
 ```python
 cors_origins: list[str] = ["http://your-organizr-domain.com"]
 ```
 **Option 2**: Proxy the API through same domain using Nginx
 ### Services Not Appearing
 **Check**:
 1. Services are deployed as Portainer stacks
 2. Services have proper labels: `com.docker.compose.project`
 3. Core-API can connect to Portainer
 4. Check browser console for errors
 ### Buttons Disabled
 **Expected Behavior**:
 - Start button disabled when service is running
 - Stop button disabled when service is stopped
 - All buttons disabled for always-on services
 ## Service Groups
 The following service groups are defined (stopping one stops all in group):
 - **jellyfin**: jellyfin
 - **nextcloud**: nextcloud, nextcloud-db, nextcloud-redis
 - **gitea**: gitea, gitea-db
 - **ai-stack**: open-webui, ollama, qdrant
 - **samba**: samba
 ## Always-On Services (Cannot be stopped)
 These infrastructure services are protected:
 - portainer
 - nginx-proxy-manager
 - core-api
 - uptime-kuma
 - organizr
 - headscale
 - watchtower
 - netdata
 - maintenance
 ## Advanced: Customizing the UI
 ### Colors
 Edit the CSS variables in the `<style>` section:
 ```css
 .status-running {
    background: rgba(72, 187, 120, 0.2);  /* Green background */
    color: #48bb78;  /* Green text */
 }
 ```
 ### Card Size
 Adjust grid columns:
 ```css
 .service-grid {
    grid-template-columns: repeat(auto-fill, minmax(300px, 1fr));
    /* Change 300px to make cards wider/narrower */
 }
 ```
 ## API Endpoints Used
 - `GET /infrastructure/services` - Fetch service list with status
 - `GET /infrastructure/service-groups` - Fetch service groups and always-on list
 - `POST /infrastructure/services/{name}/start` - Start a service
 - `POST /infrastructure/services/{name}/stop` - Stop a service
 ## Support
 For issues or questions:
 1. Check the core-api logs: `docker logs core-api`
 2. Check browser console for JavaScript errors
 3. Verify API endpoints work: `curl http://localhost:8083/infrastructure/services`
 ## License
 Part of the portainer-core project.
@@ -0,0 +1,342 @@
 <!DOCTYPE html>
 <html>
 <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Service Control</title>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }
        body {
            font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
            background: transparent;
            color: #e0e0e0;
            padding: 10px;
        }
        .container {
            max-width: 1200px;
            margin: 0 auto;
        }
        h2 {
            color: #fff;
            margin-bottom: 15px;
            font-size: 20px;
            font-weight: 500;
        }
        .service-grid {
            display: grid;
            grid-template-columns: repeat(auto-fill, minmax(300px, 1fr));
            gap: 15px;
        }
        .service-card {
            background: rgba(40, 40, 40, 0.95);
            border: 1px solid rgba(255, 255, 255, 0.1);
            border-radius: 8px;
            padding: 15px;
            transition: all 0.3s ease;
        }
        .service-card:hover {
            border-color: rgba(66, 153, 225, 0.5);
            box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
        }
        .service-header {
            display: flex;
            justify-content: space-between;
            align-items: center;
            margin-bottom: 12px;
        }
        .service-name {
            font-size: 16px;
            font-weight: 600;
            color: #fff;
            text-transform: capitalize;
        }
        .status-badge {
            padding: 4px 12px;
            border-radius: 12px;
            font-size: 12px;
            font-weight: 600;
            text-transform: uppercase;
        }
        .status-running {
            background: rgba(72, 187, 120, 0.2);
            color: #48bb78;
            border: 1px solid rgba(72, 187, 120, 0.4);
        }
        .status-stopped {
            background: rgba(245, 101, 101, 0.2);
            color: #f56565;
            border: 1px solid rgba(245, 101, 101, 0.4);
        }
        .status-loading {
            background: rgba(237, 137, 54, 0.2);
            color: #ed8936;
            border: 1px solid rgba(237, 137, 54, 0.4);
        }
        .service-info {
            font-size: 13px;
            color: #a0a0a0;
            margin-bottom: 12px;
        }
        .service-actions {
            display: flex;
            gap: 8px;
        }
        .btn {
            flex: 1;
            padding: 8px 12px;
            border: none;
            border-radius: 6px;
            font-size: 13px;
            font-weight: 600;
            cursor: pointer;
            transition: all 0.2s ease;
            text-transform: uppercase;
            letter-spacing: 0.5px;
        }
        .btn:disabled {
            opacity: 0.5;
            cursor: not-allowed;
        }
        .btn-start {
            background: linear-gradient(135deg, #48bb78 0%, #38a169 100%);
            color: white;
        }
        .btn-start:hover:not(:disabled) {
            background: linear-gradient(135deg, #38a169 0%, #2f855a 100%);
            transform: translateY(-1px);
        }
        .btn-stop {
            background: linear-gradient(135deg, #f56565 0%, #e53e3e 100%);
            color: white;
        }
        .btn-stop:hover:not(:disabled) {
            background: linear-gradient(135deg, #e53e3e 0%, #c53030 100%);
            transform: translateY(-1px);
        }
        .btn-restart {
            background: linear-gradient(135deg, #4299e1 0%, #3182ce 100%);
            color: white;
        }
        .btn-restart:hover:not(:disabled) {
            background: linear-gradient(135deg, #3182ce 0%, #2c5282 100%);
            transform: translateY(-1px);
        }
        .loading {
            text-align: center;
            padding: 40px;
            color: #a0a0a0;
        }
        .error {
            background: rgba(245, 101, 101, 0.1);
            border: 1px solid rgba(245, 101, 101, 0.4);
            color: #f56565;
            padding: 12px;
            border-radius: 6px;
            margin-bottom: 15px;
        }
        .always-on-badge {
            display: inline-block;
            padding: 2px 8px;
            background: rgba(66, 153, 225, 0.2);
            color: #4299e1;
            border: 1px solid rgba(66, 153, 225, 0.4);
            border-radius: 10px;
            font-size: 11px;
            margin-left: 8px;
        }
        @keyframes spin {
            to { transform: rotate(360deg); }
        }
        .spinner {
            display: inline-block;
            width: 14px;
            height: 14px;
            border: 2px solid rgba(255, 255, 255, 0.3);
            border-top-color: #fff;
            border-radius: 50%;
            animation: spin 0.6s linear infinite;
            margin-right: 6px;
        }
    </style>
 </head>
 <body>
    <div class="container">
        <h2>🎛️ On-Demand Services</h2>
        <div id="error-container"></div>
        <div id="service-container" class="loading">Loading services...</div>
    </div>
    <script>
        const API_BASE = 'http://localhost:8083';
        let services = [];
        let alwaysOnServices = [];
        async function fetchServices() {
            try {
                const response = await fetch(`${API_BASE}/infrastructure/services`);
                if (!response.ok) throw new Error('Failed to fetch services');
                services = await response.json();
                const groupsResponse = await fetch(`${API_BASE}/infrastructure/service-groups`);
                if (groupsResponse.ok) {
                    const groupsData = await groupsResponse.json();
                    alwaysOnServices = groupsData.always_on || [];
                }
                renderServices();
                document.getElementById('error-container').innerHTML = '';
            } catch (error) {
                console.error('Error fetching services:', error);
                document.getElementById('error-container').innerHTML =
                    `<div class="error">❌ Failed to connect to API: ${error.message}</div>`;
            }
        }
        function isAlwaysOn(serviceName) {
            return alwaysOnServices.includes(serviceName.toLowerCase());
        }
        function getServiceStatus(service) {
            if (service.containers_running > 0) {
                return {
                    class: 'status-running',
                    text: `Running (${service.containers_running}/${service.containers_total})`
                };
            } else if (service.containers_total > 0) {
                return {
                    class: 'status-stopped',
                    text: 'Stopped'
                };
            } else {
                return {
                    class: 'status-stopped',
                    text: 'No containers'
                };
            }
        }
        function renderServices() {
            const container = document.getElementById('service-container');
            // Filter to only show stoppable services
            const stoppableServices = services.filter(s => !isAlwaysOn(s.name));
            if (stoppableServices.length === 0) {
                container.innerHTML = '<div class="loading">No stoppable services found</div>';
                return;
            }
            container.className = 'service-grid';
            container.innerHTML = stoppableServices.map(service => {
                const status = getServiceStatus(service);
                const isRunning = service.containers_running > 0;
                const alwaysOn = isAlwaysOn(service.name);
                return `
                    <div class="service-card" data-service="${service.name}">
                        <div class="service-header">
                            <span class="service-name">
                                ${service.name}
                                ${alwaysOn ? '<span class="always-on-badge">ALWAYS ON</span>' : ''}
                            </span>
                            <span class="status-badge ${status.class}">${status.text}</span>
                        </div>
                        <div class="service-info">
                            Stack ID: ${service.stack_id || 'N/A'}
                        </div>
                        <div class="service-actions">
                            <button class="btn btn-start"
                                    onclick="controlService('${service.name}', 'start')"
                                    ${isRunning || alwaysOn ? 'disabled' : ''}>
                                Start
                            </button>
                            <button class="btn btn-stop"
                                    onclick="controlService('${service.name}', 'stop')"
                                    ${!isRunning || alwaysOn ? 'disabled' : ''}>
                                Stop
                            </button>
                        </div>
                    </div>
                `;
            }).join('');
        }
        async function controlService(serviceName, action) {
            const card = document.querySelector(`[data-service="${serviceName}"]`);
            const buttons = card.querySelectorAll('button');
            // Disable all buttons and show loading
            buttons.forEach(btn => {
                btn.disabled = true;
                if (btn.textContent.toLowerCase().includes(action)) {
                    btn.innerHTML = `<span class="spinner"></span>${action.toUpperCase()}...`;
                }
            });
            try {
                const response = await fetch(`${API_BASE}/infrastructure/services/${serviceName}/${action}`, {
                    method: 'POST'
                });
                const result = await response.json();
                if (!response.ok || !result.success) {
                    throw new Error(result.message || result.detail || 'Operation failed');
                }
                console.log(`${action} ${serviceName}:`, result);
                // Wait a bit for containers to start/stop
                await new Promise(resolve => setTimeout(resolve, 2000));
                // Refresh service list
                await fetchServices();
            } catch (error) {
                console.error(`Error ${action}ing ${serviceName}:`, error);
                alert(`Failed to ${action} ${serviceName}: ${error.message}`);
                // Re-enable buttons on error
                await fetchServices();
            }
        }
        // Auto-refresh every 10 seconds
        setInterval(fetchServices, 10000);
        // Initial load
        fetchServices();
    </script>
 </body>
 </html>
@@ -3,3 +3,32 @@
 # Uptime Kuma API client for automated monitor setup
 uptime-kuma-api==1.2.1
 # FastAPI and ASGI server
 fastapi==0.115.0
 uvicorn[standard]==0.32.0
 pydantic==2.10.4
 pydantic-settings==2.7.0
 # HTTP client
 httpx==0.28.1
 python-socketio[asyncio_client]==5.11.4
 # Web scraping
 beautifulsoup4==4.12.3
 trafilatura==1.12.2
 lxml==5.3.0
 # Utilities
 python-multipart==0.0.12
 python-dotenv==1.0.1
 python-json-logger==2.0.7
 # Authentication & Security
 PyJWT[crypto]==2.9.0
 python-jose[cryptography]==3.3.0
 cryptography==43.0.3
 # Memory & Embeddings
 qdrant-client==1.11.3
 sentence-transformers==3.3.1
@@ -74,7 +74,7 @@ src/
 - [x] GET /infrastructure/health - Check connectivity ✅ TESTED
 - [x] GET /infrastructure/services - List all services ✅ TESTED
 - [x] GET /infrastructure/services/{name} - Get service details ✅ TESTED
- [x] GET /infrastructure/ports - List allocated ports (skeleton)
+- [x] GET /infrastructure/ports - List allocated ports ✅ IMPLEMENTED & TESTED
 - [x] GET /infrastructure/domains - List configured domains ✅ TESTED
 - [x] Integrate with main.py routing ✅ TESTED
 - [x] Fix Pydantic validation issues (status field type conversion)
@@ -84,17 +84,27 @@ src/
 - [x] POST /infrastructure/proxy - Create NPM proxy host with optional SSL ✅ IMPLEMENTED
 - [ ] POST /infrastructure/monitoring/add - Auto-add Kuma monitor (DEFERRED - Socket.IO complexity)
-### Phase 4: Refactor Existing Controllers 📋 PENDING
+### Phase 4: Refactor Existing Controllers ✅ COMPLETE
- [ ] Move AI endpoints to ai_controller.py
+- [x] Move AI endpoints to ai_controller.py ✅ COMPLETE
- [ ] Move webscraper to tools_controller.py
+- [x] Move webscraper to tools_controller.py ✅ COMPLETE
- [ ] Move health check to health_controller.py
+- [x] Move health check to health_controller.py ✅ COMPLETE
- [ ] Update main.py imports and routing
+- [x] Update main.py imports and routing ✅ COMPLETE
 - [x] Test all refactored endpoints ✅ ALL WORKING
-### Phase 5: Testing & Documentation 📋 PENDING
+### Phase 5: Testing & Documentation ✅ COMPLETE
- [ ] Test all refactored endpoints
+- [x] Test all refactored endpoints ✅ ALL WORKING
- [ ] Update API documentation
+- [x] Update API documentation (OpenAPI spec auto-generated and validated)
- [ ] Create CLI wrapper scripts
+- [~] Create CLI wrapper scripts (SKIPPED - LLMs consume OpenAPI spec directly)
- [ ] Remove old shell scripts
+- [~] Remove old shell scripts (DEFERRED - not blocking)
 ### Phase 6: Infrastructure Improvements 📋 FUTURE
 - [ ] Consolidate Docker network topology into single `docker-dataplane` network
  - Currently each stack has its own network (172.22.0.x, 172.25.0.x, 172.20.0.x, etc.)
  - Error-prone and unnecessarily complex
  - Single shared network simplifies inter-service communication
  - Reduces subnet conflicts and improves service discovery
  - Update all compose files to use: `networks: [docker-dataplane]`
  - Create network once: `docker network create docker-dataplane`
 ---
@@ -132,7 +142,7 @@ src/
 - ✅ GET /infrastructure/services - Returns 8 active stacks
 - ✅ GET /infrastructure/services/{name} - Service lookup working
 - ✅ GET /infrastructure/domains - Returns proxy hosts with SSL status
- ⚠️ NPM health check shows `false` (returns 302 redirect instead of 200)
+- ✅ NPM health check fixed (now accepts 2xx/3xx status codes and follows redirects)
 ### Session 3: Write Endpoints (2025-11-14 Evening)
 **Completed:**
@@ -151,10 +161,79 @@ src/
 - ✅ POST /infrastructure/proxy - Implemented (not tested to avoid production interference)
 **Next Steps:**
-1. Refactor existing AI/tools/health endpoints into separate controllers (Phase 4)
+1. ~~Refactor existing AI/tools/health endpoints into separate controllers (Phase 4)~~ ✅ DONE (2025-11-14)
-2. Fix NPM health check to handle redirects
+2. ~~Fix NPM health check to handle redirects~~ ✅ DONE (2025-11-14)
-3. Implement port allocation detection logic
+3. ~~Implement port allocation detection logic~~ ✅ DONE (2025-11-14)
-4. Create CLI wrappers for common operations
+4. ~~Create CLI wrappers for common operations~~ ⊘ SKIPPED (LLMs use OpenAPI)
 5. (OPTIONAL) Consolidate Docker networks into `docker-dataplane` (Phase 6)
 ### Session 4: NPM Health Check & Port Detection (2025-11-14 Afternoon)
 **Completed:**
 - Fixed NPM health check to handle redirects properly
 - Updated `npm_client.py` to accept 2xx/3xx status codes as healthy
 - Enabled explicit redirect following in httpx client
 - Verified fix with live NPM instance (now shows 9 proxy hosts)
 - Implemented comprehensive port detection in `GET /infrastructure/ports` endpoint
 - Added `get_containers()` and `get_container()` methods to PortainerClient
 - Enhanced PortInfo model with internal/external hostname and IP fields
 - Implemented domain mapping from NPM proxy hosts to services
 - Added deduplication logic for port entries (Docker returns duplicates per bind address)
 **Port Detection Features:**
 - Scans all running containers across all Portainer endpoints
 - Extracts internal port, host port, and protocol for each container
 - Maps container names to service names via Docker Compose labels
 - Retrieves internal Docker hostnames and IP addresses per network
 - Cross-references NPM proxy hosts to identify external domains
 - Returns 22 unique port mappings with complete metadata
 **Technical Details:**
 *NPM Health Check:*
 - Issue: NPM's `/api` endpoint returns 302 redirect, old code only accepted 200
 - Solution: Accept `200 <= status_code < 400` as healthy response
 - Result: NPM health check now returns `true` and proxy hosts are enumerated correctly
 *Port Detection:*
 - Queries Portainer Docker API for container list and port mappings
 - Extracts NetworkSettings for internal IPs and hostnames
 - Builds port→domain map from NPM proxy hosts configuration
 - Matches services to external domains using multiple strategies:
  - By container name + port
  - By internal IP + port
  - By host address + host port (localhost, 127.0.0.1, server IP)
 - Deduplicates based on (port, container_name, protocol) tuple
 - Example output: Nextcloud port 80 → internal IP 172.25.0.3 → external domain cloud.schweitz.net
 ### Session 5: Controller Architecture Refactoring (2025-11-14 Evening)
 **Completed:**
 - Created `ai_controller.py` consolidating chat, models, and conversations endpoints
 - Created `tools_controller.py` for web scraper functionality
 - Created `health_controller.py` for service health and info endpoints
 - Updated `main.py` to use new controller-based architecture
 - Removed legacy router imports and inline endpoint definitions
 - Tested all refactored endpoints - 16 endpoints working correctly
 **Architecture Changes:**
 - All endpoints now follow consistent controller pattern inheriting from `BaseController`
 - Controllers use `create_router()` method for FastAPI router configuration
 - Clean separation of concerns:
  - `ai_controller.py` - AI orchestration and conversation memory (7 endpoints)
  - `tools_controller.py` - Utility tools like web scraper (1 endpoint)
  - `health_controller.py` - Service status and info (2 endpoints)
  - `infrastructure_controller.py` - Infrastructure management (6 endpoints)
 - Simplified `main.py` from 220 lines to 152 lines
 - Backward compatible - all existing endpoints work identically
 **Test Results:**
 - ✅ GET / - Service information
 - ✅ GET /health - Health check with Ollama status
 - ✅ GET /v1/models - Model listing
 - ✅ POST /v1/chat/completions - Chat completions
 - ✅ GET /v1/conversations/{id} - Conversation history
 - ✅ GET /infrastructure/health - Infrastructure health
 - ✅ POST /web-scraper/scrape - Web scraping
 - ✅ OpenAPI spec generation - 16 endpoints documented
 ## API Authentication Strategy
@@ -0,0 +1,152 @@
 #!/usr/bin/env python3
 """
 Create Authentik Outpost for NPM Forward Authentication
 Creates a dedicated outpost and retrieves its token for deployment.
 """
 import asyncio
 import sys
 sys.path.insert(0, '/home/jpmschweitzer/Projects/portainer-core/services/core-api')
 from src.clients.authentik_client import get_authentik_client
 async def create_npm_outpost():
    """Create outpost for NPM forward auth and get token"""
    authentik = get_authentik_client()
    try:
        print("=" * 60)
        print("Creating NPM Forward Auth Outpost")
        print("=" * 60)
        # Check if provider exists
        print("\n1. Checking for proxy provider...")
        provider = await authentik.get_provider_by_name_proxy("npm-forward-auth-provider")
        if not provider:
            print("❌ Proxy provider not found. Run setup_authentik_forward_auth.py first.")
            return False
        provider_id = provider["pk"]
        print(f"✓ Found provider (ID: {provider_id})")
        # Check if outpost already exists
        print("\n2. Checking for existing NPM outpost...")
        try:
            existing = await authentik.get_outpost_by_name("npm-forward-auth-outpost")
            if existing:
                print(f"✓ Outpost already exists (ID: {existing['pk']})")
                outpost_id = existing["pk"]
                # Update it to ensure provider is assigned
                print("\n3. Updating outpost configuration...")
                await authentik.update_outpost(
                    outpost_id=outpost_id,
                    providers=[provider_id]
                )
                print("✓ Outpost updated with provider")
        except Exception:
            # Outpost doesn't exist, create it
            print("⊘ Outpost doesn't exist, creating new one...")
            print("\n3. Creating NPM outpost...")
            outpost = await authentik.create_outpost(
                name="npm-forward-auth-outpost",
                type="proxy",
                providers=[provider_id],
                config={
                    "authentik_host": "http://192.168.86.149:9000",
                    "authentik_host_insecure": False,
                    "log_level": "info",
                    "docker_labels": None,
                    "docker_network": None,
                    "docker_map_ports": True,
                    "container_image": None,
                    "kubernetes_replicas": 1,
                    "kubernetes_namespace": "default"
                }
            )
            outpost_id = outpost["pk"]
            print(f"✓ Created outpost (ID: {outpost_id})")
        # Try to get the service connection token
        print("\n4. Retrieving outpost token...")
        print("\nNote: Authentik creates service accounts for outposts automatically.")
        print("The token format is: ak-outpost-<outpost_uuid>-api")
        # Get outpost details to find its service account
        outpost_details = await authentik._request("GET", f"outposts/instances/{outpost_id}/")
        print(f"\nOutpost Details:")
        print(f"  Name: {outpost_details.get('name')}")
        print(f"  ID: {outpost_details.get('pk')}")
        print(f"  Type: {outpost_details.get('type')}")
        print(f"  Providers: {outpost_details.get('providers')}")
        # The outpost service connection details
        if 'service_connection' in outpost_details:
            print(f"  Service Connection: {outpost_details.get('service_connection')}")
        # List tokens to find the one for this outpost
        print("\n5. Looking for outpost service token...")
        tokens = await authentik.list_tokens()
        outpost_uuid = outpost_details.get('pk')
        token_identifier = f"ak-outpost-{outpost_uuid}-api"
        matching_token = None
        for token in tokens:
            if token.get('identifier') == token_identifier:
                matching_token = token
                break
        if matching_token:
            print(f"✓ Found token: {matching_token.get('identifier')}")
            print(f"\n{'=' * 60}")
            print("IMPORTANT: Token Key Required")
            print("=" * 60)
            print("\nAuthentik does not expose token keys via API after creation.")
            print("\nTo get the token key:")
            print("1. Go to: https://auth.schweitz.net/if/admin/#/core/tokens")
            print(f"2. Find token: {token_identifier}")
            print("3. Click 'View Token Key' or regenerate the token")
            print("4. Copy the token key")
            print("\nAlternatively, you can:")
            print("1. Delete the existing outpost via UI")
            print("2. Create a new outpost via UI")
            print("3. Copy the token key when it's displayed")
            print("\n" + "=" * 60)
        else:
            print("\n⚠ No automatic token found.")
            print("You may need to manually create a token for the outpost.")
            print("\nManual token creation:")
            print("1. Go to: https://auth.schweitz.net/if/admin/#/core/tokens")
            print("2. Click 'Create'")
            print(f"3. Identifier: npm-outpost-token")
            print("4. User: Select the outpost service account")
            print("5. Intent: API")
            print("6. Copy the token key when displayed")
        print("\n" + "=" * 60)
        print("Next Steps")
        print("=" * 60)
        print("\n1. Obtain the outpost token key (see above)")
        print("2. Create/update .env.authentik-shared file:")
        print("   AUTHENTIK_OUTPOST_TOKEN=<your_token_key>")
        print("3. Deploy the stack:")
        print("   docker-compose -f stacks/authentik-shared.yml --env-file .env.authentik-shared up -d")
        print("4. Update NPM hosts to point to port 9001")
        return True
    except Exception as e:
        print(f"\n❌ Error: {e}")
        import traceback
        traceback.print_exc()
        return False
 if __name__ == "__main__":
    success = asyncio.run(create_npm_outpost())
    sys.exit(0 if success else 1)
@@ -6,6 +6,7 @@ pydantic-settings==2.7.0
 # HTTP client
 httpx==0.28.1
 python-socketio[asyncio_client]==5.11.4
 # Web scraping
 beautifulsoup4==4.12.3
@@ -17,6 +18,11 @@ python-multipart==0.0.12
 python-dotenv==1.0.1
 python-json-logger==2.0.7
 # Authentication & Security
 PyJWT[crypto]==2.9.0
 python-jose[cryptography]==3.3.0
 cryptography==43.0.3
 # Memory & Embeddings
 qdrant-client==1.11.3
 sentence-transformers==3.3.1
@@ -0,0 +1,118 @@
 #!/usr/bin/env python3
 """
 Setup Authentik Forward Authentication for NPM
 This script creates a proxy provider and outpost for forward authentication
 across all NPM-managed domains.
 """
 import asyncio
 import sys
 sys.path.insert(0, '/home/jpmschweitzer/Projects/portainer-core/services/core-api')
 from src.clients.authentik_client import get_authentik_client
 async def setup_forward_auth():
    """Create proxy provider, application, and outpost for forward auth"""
    client = get_authentik_client()
    try:
        # Check if Authentik is accessible
        print("Checking Authentik connectivity...")
        if not await client.health_check():
            print("❌ Authentik is not accessible")
            return False
        print("✓ Authentik is accessible")
        # Check if proxy provider already exists
        print("\nChecking for existing proxy provider...")
        existing_provider = await client.get_provider_by_name_proxy("npm-forward-auth-provider")
        if existing_provider:
            print(f"✓ Proxy provider already exists (ID: {existing_provider.get('pk')})")
            provider = existing_provider
        else:
            # Create Proxy provider for forward auth
            print("\nCreating Proxy provider for forward authentication...")
            provider = await client.create_proxy_provider(
                name="npm-forward-auth-provider",
                external_host="https://auth.schweitz.net",
                mode="forward_single",
                token_validity=480  # 8 hours
            )
            print(f"✓ Created proxy provider (ID: {provider.get('pk')})")
        # Display provider details
        print("\n" + "="*60)
        print("Proxy Provider Details:")
        print("="*60)
        print(f"Provider ID: {provider.get('pk')}")
        print(f"Mode: {provider.get('mode')}")
        print(f"External Host: {provider.get('external_host')}")
        print(f"Token Validity: {provider.get('access_token_validity')}")
        print(f"Session Duration: {provider.get('session_duration')}")
        print("="*60)
        # Check if application already exists
        print("\nChecking for existing application...")
        existing_app = await client.get_application_by_slug("npm-forward-auth")
        if existing_app:
            print(f"✓ Application already exists (slug: {existing_app.get('slug')})")
            app = existing_app
        else:
            # Create application
            print("\nCreating application...")
            app = await client.create_application(
                name="NPM Forward Auth",
                slug="npm-forward-auth",
                provider_pk=provider.get("pk"),
                launch_url="https://auth.schweitz.net"
            )
            print(f"✓ Created application (slug: {app.get('slug')})")
        # Check if outpost already exists
        print("\nChecking for existing outpost...")
        existing_outpost = await client.get_outpost_by_name("npm-forward-auth-outpost")
        if existing_outpost:
            print(f"✓ Outpost already exists (ID: {existing_outpost.get('pk')})")
            outpost = existing_outpost
        else:
            # Create outpost
            print("\nCreating outpost...")
            outpost = await client.create_outpost(
                name="npm-forward-auth-outpost",
                type="proxy",
                providers=[provider.get("pk")],
                config={
                    "authentik_host": "https://auth.schweitz.net",
                    "authentik_host_insecure": False,
                    "log_level": "info"
                }
            )
            print(f"✓ Created outpost (ID: {outpost.get('pk')})")
        print("\n" + "="*60)
        print("Setup Complete!")
        print("="*60)
        print("\nNext steps:")
        print("1. Deploy the Authentik outpost container")
        print("2. Configure NPM proxy hosts with forward auth")
        print("3. Test the SSO flow")
        print("\nOutpost deployment command will be generated...")
        return True
    except Exception as e:
        print(f"\n❌ Error: {e}")
        import traceback
        traceback.print_exc()
        return False
    finally:
        await client.close()
 if __name__ == "__main__":
    success = asyncio.run(setup_forward_auth())
    sys.exit(0 if success else 1)
@@ -0,0 +1,5 @@
 """
 Authentication module for core-api
 Provides OIDC/OAuth2 authentication via Authentik
 """
@@ -0,0 +1,235 @@
 """
 OIDC Authentication Module
 Provides OAuth2/OIDC token validation for FastAPI using Authentik as IdP.
 Implements bearer token authentication with JWT verification.
 """
 from fastapi import Depends, HTTPException, Security
 from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
 from jose import jwt, JWTError
 import httpx
 from functools import lru_cache
 from typing import Dict, Optional
 from src.logging_config import get_logger
 logger = get_logger(__name__)
 security = HTTPBearer(auto_error=False)
 class OIDCConfig:
    """OIDC configuration from environment"""
    def __init__(self):
        # These will be set from environment variables in config.py
        self.enabled = False
        self.issuer = ""
        self.audience = ""
        self.jwks_uri = ""
    def configure(self, enabled: bool, issuer: str, audience: str):
        """Configure OIDC settings"""
        self.enabled = enabled
        self.issuer = issuer
        self.audience = audience
        self.jwks_uri = f"{issuer.rstrip('/')}/jwks/"
        logger.info(f"OIDC configured: enabled={enabled}, issuer={issuer}")
 # Global OIDC config instance
 oidc_config = OIDCConfig()
@lru_cache(maxsize=1)
 def get_jwks() -> Dict:
    """
    Fetch JSON Web Key Set (JWKS) from Authentik
    Cached to avoid repeated requests. Cache is cleared on server restart.
    Returns:
        JWKS dictionary containing public keys for token verification
    Raises:
        HTTPException: If JWKS fetch fails
    """
    if not oidc_config.enabled:
        return {}
    try:
        logger.debug(f"Fetching JWKS from {oidc_config.jwks_uri}")
        response = httpx.get(oidc_config.jwks_uri, timeout=10.0)
        response.raise_for_status()
        jwks = response.json()
        logger.info(f"JWKS fetched successfully ({len(jwks.get('keys', []))} keys)")
        return jwks
    except Exception as e:
        logger.error(f"Failed to fetch JWKS: {e}")
        raise HTTPException(
            status_code=503,
            detail="Authentication service unavailable"
        )
 async def get_current_user(
    credentials: Optional[HTTPAuthorizationCredentials] = Security(security)
 ) -> Optional[Dict]:
    """
    Validate OIDC token from Authorization: Bearer header
    Extracts and validates JWT token from request header. Verifies:
    - Token signature using JWKS
    - Token expiration
    - Issuer matches Authentik
    - Audience matches core-api
    Args:
        credentials: HTTP Bearer token from Authorization header
    Returns:
        User claims dictionary containing email, name, groups, etc.
        Returns None if OIDC is disabled (allows unauthenticated access)
    Raises:
        HTTPException 401: If token is invalid, expired, or missing when OIDC enabled
    """
    # If OIDC is disabled, allow all requests (no authentication)
    if not oidc_config.enabled:
        logger.debug("OIDC disabled - allowing unauthenticated access")
        return None
    # OIDC enabled - token required
    if not credentials:
        logger.warning("Authentication required but no token provided")
        raise HTTPException(
            status_code=401,
            detail="Authentication required",
            headers={"WWW-Authenticate": "Bearer"},
        )
    token = credentials.credentials
    try:
        # Decode token header to get key ID
        unverified_header = jwt.get_unverified_header(token)
        kid = unverified_header.get("kid")
        if not kid:
            raise HTTPException(status_code=401, detail="Invalid token format")
        # Find matching key in JWKS
        jwks = get_jwks()
        rsa_key = None
        for key in jwks.get("keys", []):
            if key.get("kid") == kid:
                rsa_key = key
                break
        if not rsa_key:
            logger.warning(f"No matching key found for kid: {kid}")
            raise HTTPException(status_code=401, detail="Invalid token key")
        # Verify and decode token
        payload = jwt.decode(
            token,
            rsa_key,
            algorithms=["RS256"],
            audience=oidc_config.audience,
            issuer=oidc_config.issuer,
        )
        user_email = payload.get("email", "unknown")
        logger.info(f"Authenticated user: {user_email}")
        return payload
    except jwt.ExpiredSignatureError:
        logger.warning("Token expired")
        raise HTTPException(
            status_code=401,
            detail="Token expired",
            headers={"WWW-Authenticate": "Bearer"},
        )
    except jwt.JWTClaimsError as e:
        logger.warning(f"Invalid token claims: {e}")
        raise HTTPException(
            status_code=401,
            detail="Invalid token claims",
            headers={"WWW-Authenticate": "Bearer"},
        )
    except JWTError as e:
        logger.error(f"JWT validation error: {e}")
        raise HTTPException(
            status_code=401,
            detail="Invalid authentication token",
            headers={"WWW-Authenticate": "Bearer"},
        )
    except Exception as e:
        logger.error(f"Unexpected authentication error: {e}")
        raise HTTPException(
            status_code=500,
            detail="Authentication error",
        )
 async def get_admin_user(
    user: Optional[Dict] = Depends(get_current_user)
 ) -> Dict:
    """
    Require admin group membership
    Use this dependency for endpoints that require admin access.
    Checks if user is member of 'admin' group in Authentik.
    Args:
        user: User claims from get_current_user
    Returns:
        User claims dictionary if user is admin
    Raises:
        HTTPException 403: If user is not in admin group
        HTTPException 401: If OIDC enabled but user not authenticated
    """
    # If OIDC disabled, allow all (backward compatibility)
    if not oidc_config.enabled or user is None:
        logger.debug("OIDC disabled - allowing admin access")
        return {"email": "unauthenticated", "groups": ["admin"]}
    # Check admin group membership
    groups = user.get("groups", [])
    if "admin" not in groups and "authentik Admins" not in groups:
        user_email = user.get("email", "unknown")
        logger.warning(f"User {user_email} attempted admin access (groups: {groups})")
        raise HTTPException(
            status_code=403,
            detail="Admin access required"
        )
    return user
 async def get_optional_user(
    credentials: Optional[HTTPAuthorizationCredentials] = Security(security)
 ) -> Optional[Dict]:
    """
    Optional authentication - allows both authenticated and unauthenticated access
    Use for endpoints that should be accessible to everyone but can provide
    enhanced functionality for authenticated users.
    Args:
        credentials: HTTP Bearer token from Authorization header
    Returns:
        User claims if valid token provided, None otherwise
    """
    if not credentials or not oidc_config.enabled:
        return None
    try:
        return await get_current_user(credentials)
    except HTTPException:
        # Invalid token - return None instead of raising
        return None
@@ -0,0 +1,302 @@
 """
 Authentik API Client
 Provides methods for interacting with Authentik Identity Provider API.
 Used for managing applications, providers, and authentication flows.
 """
 import httpx
 from typing import Dict, List, Any, Optional
 from functools import lru_cache
 from src.logging_config import get_logger
 logger = get_logger(__name__)
 class AuthentikClient:
    """Client for Authentik API operations"""
    def __init__(self, base_url: str, api_token: str):
        """
        Initialize Authentik client
        Args:
            base_url: Authentik base URL (e.g., http://authentik-server:9000)
            api_token: API token for authentication
        """
        self.base_url = base_url.rstrip('/')
        self.api_token = api_token
        self.client = httpx.AsyncClient(timeout=30.0)
    async def _request(self, method: str, endpoint: str, **kwargs) -> Dict:
        """Make authenticated API request using token auth"""
        headers = kwargs.pop("headers", {})
        headers["Authorization"] = f"Bearer {self.api_token}"
        response = await self.client.request(
            method,
            f"{self.base_url}/api/v3/{endpoint.lstrip('/')}",
            headers=headers,
            **kwargs
        )
        if not response.is_success:
            logger.error(f"API request failed: {response.status_code}")
            logger.error(f"Response body: {response.text}")
        response.raise_for_status()
        return response.json()
    async def health_check(self) -> bool:
        """Check if Authentik is accessible"""
        try:
            response = await self.client.get(f"{self.base_url}/-/health/live/")
            return response.status_code == 200
        except Exception as e:
            logger.error(f"Authentik health check failed: {e}")
            return False
    async def create_oauth2_provider(
        self,
        name: str,
        client_id: str,
        redirect_uris: List[str],
        authorization_flow_slug: str = "default-provider-authorization-implicit-consent",
        signing_key: Optional[str] = None
    ) -> Dict:
        """
        Create an OAuth2/OIDC provider
        Args:
            name: Provider name
            client_id: OAuth2 client ID
            redirect_uris: List of allowed redirect URIs
            authorization_flow_slug: Authorization flow slug (will be resolved to UUID)
            signing_key: Signing key UUID (defaults to auto-selected)
        Returns:
            Created provider data including client_secret
        """
        # Get authorization flow UUID from slug
        flows = await self.list_flows()
        auth_flow_uuid = None
        invalidation_flow_uuid = None
        for flow in flows:
            if flow.get("slug") == authorization_flow_slug:
                auth_flow_uuid = flow.get("pk")
            if flow.get("slug") == "default-provider-invalidation-flow":
                invalidation_flow_uuid = flow.get("pk")
        if not auth_flow_uuid:
            raise ValueError(f"Authorization flow '{authorization_flow_slug}' not found")
        if not invalidation_flow_uuid:
            raise ValueError("Invalidation flow not found")
        # Get signing key if not provided
        if not signing_key:
            keys = await self._request("GET", "crypto/certificatekeypairs/")
            # Find the self-signed cert
            for key in keys.get("results", []):
                if "authentik" in key.get("name", "").lower():
                    signing_key = key.get("pk")
                    break
            if not signing_key and keys.get("results"):
                signing_key = keys["results"][0]["pk"]
        # Format redirect URIs as objects with matching_mode
        formatted_redirect_uris = [
            {"url": uri, "matching_mode": "strict"}
            for uri in redirect_uris
        ]
        provider_data = {
            "name": name,
            "authorization_flow": auth_flow_uuid,
            "invalidation_flow": invalidation_flow_uuid,
            "client_type": "confidential",
            "client_id": client_id,
            "redirect_uris": formatted_redirect_uris,
            "signing_key": signing_key,
            "sub_mode": "hashed_user_id",
            "include_claims_in_id_token": True,
            "issuer_mode": "per_provider",
            "access_token_validity": "minutes=60",
            "refresh_token_validity": "days=30",
            "property_mappings": []  # Will use default mappings
        }
        result = await self._request("POST", "providers/oauth2/", json=provider_data)
        logger.info(f"Created OAuth2 provider: {name} (ID: {result.get('pk')})")
        return result
    async def create_application(
        self,
        name: str,
        slug: str,
        provider_pk: int,
        launch_url: Optional[str] = None,
        icon_url: Optional[str] = None
    ) -> Dict:
        """
        Create an application
        Args:
            name: Application display name
            slug: Application slug (URL-safe identifier)
            provider_pk: Primary key of the provider to use
            launch_url: Optional launch URL
            icon_url: Optional icon URL
        Returns:
            Created application data
        """
        app_data = {
            "name": name,
            "slug": slug,
            "provider": provider_pk,
            "meta_launch_url": launch_url or "",
            "meta_icon": icon_url or "",
            "policy_engine_mode": "any",
            "open_in_new_tab": False
        }
        result = await self._request("POST", "core/applications/", json=app_data)
        logger.info(f"Created application: {name} (slug: {slug})")
        return result
    async def get_provider_by_name(self, name: str) -> Optional[Dict]:
        """Get OAuth2 provider by name"""
        providers = await self._request("GET", "providers/oauth2/", params={"name": name})
        results = providers.get("results", [])
        return results[0] if results else None
    async def get_application_by_slug(self, slug: str) -> Optional[Dict]:
        """Get application by slug"""
        apps = await self._request("GET", "core/applications/", params={"slug": slug})
        results = apps.get("results", [])
        return results[0] if results else None
    async def list_flows(self) -> List[Dict]:
        """List all authentication flows"""
        result = await self._request("GET", "flows/instances/")
        return result.get("results", [])
    async def create_proxy_provider(
        self,
        name: str,
        external_host: str,
        authorization_flow_slug: str = "default-provider-authorization-implicit-consent",
        mode: str = "forward_single",
        token_validity: int = 480  # 8 hours in minutes
    ) -> Dict:
        """
        Create a Proxy Provider for forward authentication
        Args:
            name: Provider name
            external_host: External URL (e.g., https://auth.schweitz.net)
            authorization_flow_slug: Authorization flow slug
            mode: Proxy mode (forward_single for forward auth)
            token_validity: Token validity in minutes (default: 480 = 8 hours)
        Returns:
            Created provider data
        """
        # Get authorization flow UUID from slug
        flows = await self.list_flows()
        auth_flow_uuid = None
        invalidation_flow_uuid = None
        for flow in flows:
            if flow.get("slug") == authorization_flow_slug:
                auth_flow_uuid = flow.get("pk")
            if flow.get("slug") == "default-provider-invalidation-flow":
                invalidation_flow_uuid = flow.get("pk")
        if not auth_flow_uuid:
            raise ValueError(f"Authorization flow '{authorization_flow_slug}' not found")
        if not invalidation_flow_uuid:
            raise ValueError("Invalidation flow not found")
        provider_data = {
            "name": name,
            "authorization_flow": auth_flow_uuid,
            "invalidation_flow": invalidation_flow_uuid,
            "mode": mode,
            "external_host": external_host,
            "access_token_validity": f"minutes={token_validity}",
            "refresh_token_validity": f"minutes={token_validity}",
            "session_duration": f"seconds={token_validity * 60}",
            "cookie_domain": "",  # Will use the domain of each proxied site
            "property_mappings": []
        }
        result = await self._request("POST", "providers/proxy/", json=provider_data)
        logger.info(f"Created Proxy provider: {name} (ID: {result.get('pk')})")
        return result
    async def get_provider_by_name_proxy(self, name: str) -> Optional[Dict]:
        """Get Proxy provider by name"""
        providers = await self._request("GET", "providers/proxy/", params={"name": name})
        results = providers.get("results", [])
        return results[0] if results else None
    async def create_outpost(
        self,
        name: str,
        type: str,
        providers: List[int],
        config: Optional[Dict] = None
    ) -> Dict:
        """
        Create an Authentik Outpost
        Args:
            name: Outpost name
            type: Outpost type (e.g., "proxy")
            providers: List of provider PKs
            config: Optional configuration overrides
        Returns:
            Created outpost data
        """
        outpost_data = {
            "name": name,
            "type": type,
            "providers": providers,
            "config": config or {},
            "service_connection": None  # Will use local Docker
        }
        result = await self._request("POST", "outposts/instances/", json=outpost_data)
        logger.info(f"Created outpost: {name} (ID: {result.get('pk')})")
        return result
    async def get_outpost_by_name(self, name: str) -> Optional[Dict]:
        """Get outpost by name"""
        outposts = await self._request("GET", "outposts/instances/", params={"name": name})
        results = outposts.get("results", [])
        return results[0] if results else None
    async def close(self):
        """Close HTTP client"""
        await self.client.aclose()
@lru_cache()
 def get_authentik_client() -> AuthentikClient:
    """Get cached Authentik client instance"""
    # Import credentials from gitignored module
    try:
        from src.credentials import AUTHENTIK_URL, AUTHENTIK_CORE_API_TOKEN
    except ImportError:
        # Fallback to environment variables if credentials.py doesn't exist
        import os
        AUTHENTIK_URL = os.getenv("AUTHENTIK_URL", "http://authentik-server:9000")
        AUTHENTIK_CORE_API_TOKEN = os.getenv("AUTHENTIK_API_TOKEN", "")
    return AuthentikClient(
        base_url=AUTHENTIK_URL,
        api_token=AUTHENTIK_CORE_API_TOKEN
    )
@@ -0,0 +1,441 @@
 """
 Uptime Kuma Socket.IO Client
 Provides interface to Uptime Kuma via Socket.IO for monitor management.
 """
 import socketio
 import asyncio
 from typing import Optional, Dict, List, Any
 from src.logging_config import get_logger
 from src.config import get_settings
 logger = get_logger(__name__)
 settings = get_settings()
 class KumaClient:
    """
    Socket.IO client for Uptime Kuma
    Uses Socket.IO for real-time communication with Uptime Kuma.
    """
    def __init__(
        self,
        base_url: Optional[str] = None,
        username: Optional[str] = None,
        password: Optional[str] = None,
        timeout: int = 30
    ):
        """
        Initialize Kuma client
        Args:
            base_url: Kuma base URL (default from settings)
            username: Kuma username (default from settings)
            password: Kuma password (default from settings)
            timeout: Request timeout in seconds
        """
        self.base_url = (base_url or settings.kuma_url).rstrip("/")
        self.username = username or settings.kuma_username
        self.password = password or settings.kuma_password
        self.timeout = timeout
        self.sio = socketio.AsyncClient(
            reconnection=True,
            reconnection_attempts=3,
            reconnection_delay=1,
        )
        self._connected = False
        self._authenticated = False
        self._monitors_cache: Dict[int, Dict[str, Any]] = {}
        if not self.username or not self.password:
            logger.warning("Uptime Kuma credentials not configured")
    async def _ensure_connected(self):
        """Ensure we have an active connection and authentication"""
        if not self._connected:
            await self.connect()
        if not self._authenticated:
            await self.login()
    async def connect(self):
        """Connect to Uptime Kuma Socket.IO server"""
        if self._connected:
            return
        try:
            await self.sio.connect(self.base_url, transports=['websocket'])
            self._connected = True
            logger.info(f"Connected to Uptime Kuma at {self.base_url}")
        except Exception as e:
            logger.error(f"Failed to connect to Uptime Kuma: {e}")
            raise
    async def disconnect(self):
        """Disconnect from Uptime Kuma"""
        if self._connected:
            await self.sio.disconnect()
            self._connected = False
            self._authenticated = False
            logger.info("Disconnected from Uptime Kuma")
    async def login(self):
        """Authenticate with Uptime Kuma"""
        if not self._connected:
            await self.connect()
        try:
            # Uptime Kuma login event
            login_response = await self.sio.call(
                'login',
                {
                    'username': self.username,
                    'password': self.password,
                    'token': None
                },
                timeout=self.timeout
            )
            if login_response and login_response.get('ok'):
                self._authenticated = True
                logger.info("Successfully authenticated with Uptime Kuma")
            else:
                error_msg = login_response.get('msg', 'Unknown error') if login_response else 'No response'
                raise Exception(f"Login failed: {error_msg}")
        except Exception as e:
            logger.error(f"Failed to authenticate with Uptime Kuma: {e}")
            raise
    async def health_check(self) -> bool:
        """
        Check if Uptime Kuma is accessible
        Returns:
            True if accessible, False otherwise
        """
        try:
            await self._ensure_connected()
            return self._authenticated
        except Exception as e:
            logger.error(f"Uptime Kuma health check failed: {e}")
            return False
    async def get_monitors(self) -> List[Dict[str, Any]]:
        """
        List all monitors
        Returns:
            List of monitor configurations
        """
        await self._ensure_connected()
        try:
            # Get monitor list
            response = await self.sio.call('getMonitorList', timeout=self.timeout)
            if response and isinstance(response, dict):
                # Uptime Kuma returns monitors as a dict with monitor IDs as keys
                monitors = []
                for monitor_id, monitor_data in response.items():
                    if isinstance(monitor_data, dict):
                        monitor_data['id'] = int(monitor_id)
                        monitors.append(monitor_data)
                        self._monitors_cache[int(monitor_id)] = monitor_data
                return monitors
            return []
        except Exception as e:
            logger.error(f"Failed to get monitors: {e}")
            raise
    async def get_monitor(self, monitor_id: int) -> Dict[str, Any]:
        """
        Get details of a specific monitor
        Args:
            monitor_id: Monitor identifier
        Returns:
            Monitor configuration details
        """
        await self._ensure_connected()
        try:
            response = await self.sio.call('getMonitor', monitor_id, timeout=self.timeout)
            if response:
                self._monitors_cache[monitor_id] = response
                return response
            raise Exception(f"Monitor {monitor_id} not found")
        except Exception as e:
            logger.error(f"Failed to get monitor {monitor_id}: {e}")
            raise
    async def find_monitor_by_name(self, name: str) -> Optional[Dict[str, Any]]:
        """
        Find a monitor by its name (case-insensitive)
        Args:
            name: Monitor name to search for
        Returns:
            Monitor object if found, None otherwise
        """
        monitors = await self.get_monitors()
        name_lower = name.lower()
        for monitor in monitors:
            if monitor.get("name", "").lower() == name_lower:
                return monitor
        return None
    async def find_monitors_by_tag(self, tag: str) -> List[Dict[str, Any]]:
        """
        Find all monitors with a specific tag
        Args:
            tag: Tag name to search for
        Returns:
            List of monitors with the tag
        """
        monitors = await self.get_monitors()
        tagged_monitors = []
        for monitor in monitors:
            monitor_tags = monitor.get("tags", [])
            if any(t.get("name", "").lower() == tag.lower() for t in monitor_tags):
                tagged_monitors.append(monitor)
        return tagged_monitors
    async def pause_monitor(self, monitor_id: int) -> bool:
        """
        Pause a monitor (disable monitoring)
        Args:
            monitor_id: Monitor identifier
        Returns:
            True if successful
        """
        await self._ensure_connected()
        try:
            # Uptime Kuma pause event
            response = await self.sio.call('pauseMonitor', monitor_id, timeout=self.timeout)
            if response and response.get('ok'):
                logger.info(f"Paused monitor {monitor_id}")
                return True
            error_msg = response.get('msg', 'Unknown error') if response else 'No response'
            raise Exception(f"Failed to pause monitor: {error_msg}")
        except Exception as e:
            logger.error(f"Failed to pause monitor {monitor_id}: {e}")
            raise
    async def resume_monitor(self, monitor_id: int) -> bool:
        """
        Resume a monitor (enable monitoring)
        Args:
            monitor_id: Monitor identifier
        Returns:
            True if successful
        """
        await self._ensure_connected()
        try:
            # Uptime Kuma resume event
            response = await self.sio.call('resumeMonitor', monitor_id, timeout=self.timeout)
            if response and response.get('ok'):
                logger.info(f"Resumed monitor {monitor_id}")
                return True
            error_msg = response.get('msg', 'Unknown error') if response else 'No response'
            raise Exception(f"Failed to resume monitor: {error_msg}")
        except Exception as e:
            logger.error(f"Failed to resume monitor {monitor_id}: {e}")
            raise
    async def pause_monitor_by_name(self, name: str) -> bool:
        """
        Pause a monitor by its name
        Args:
            name: Monitor name
        Returns:
            True if successful, False if monitor not found
        """
        monitor = await self.find_monitor_by_name(name)
        if not monitor:
            logger.warning(f"Monitor '{name}' not found")
            return False
        await self.pause_monitor(monitor["id"])
        return True
    async def resume_monitor_by_name(self, name: str) -> bool:
        """
        Resume a monitor by its name
        Args:
            name: Monitor name
        Returns:
            True if successful, False if monitor not found
        """
        monitor = await self.find_monitor_by_name(name)
        if not monitor:
            logger.warning(f"Monitor '{name}' not found")
            return False
        await self.resume_monitor(monitor["id"])
        return True
    async def add_monitor(self, monitor_config: Dict[str, Any]) -> Dict[str, Any]:
        """
        Create a new monitor
        Args:
            monitor_config: Monitor configuration dict
        Returns:
            Created monitor details including ID
        """
        await self._ensure_connected()
        try:
            # Uptime Kuma add monitor event
            response = await self.sio.call('add', monitor_config, timeout=self.timeout)
            if response and response.get('ok'):
                monitor_id = response.get('monitorID')
                logger.info(f"Created monitor '{monitor_config.get('name')}' with ID {monitor_id}")
                # Get full monitor details
                monitor = await self.get_monitor(monitor_id)
                return monitor
            error_msg = response.get('msg', 'Unknown error') if response else 'No response'
            raise Exception(f"Failed to create monitor: {error_msg}")
        except Exception as e:
            logger.error(f"Failed to create monitor '{monitor_config.get('name')}': {e}")
            raise
    async def update_monitor(self, monitor_id: int, monitor_config: Dict[str, Any]) -> Dict[str, Any]:
        """
        Update an existing monitor
        Args:
            monitor_id: Monitor identifier
            monitor_config: Updated monitor configuration
        Returns:
            Updated monitor details
        """
        await self._ensure_connected()
        try:
            # Ensure ID is in the config
            monitor_config['id'] = monitor_id
            # Uptime Kuma edit monitor event
            response = await self.sio.call('editMonitor', monitor_config, timeout=self.timeout)
            if response and response.get('ok'):
                logger.info(f"Updated monitor {monitor_id}")
                # Get updated monitor details
                monitor = await self.get_monitor(monitor_id)
                return monitor
            error_msg = response.get('msg', 'Unknown error') if response else 'No response'
            raise Exception(f"Failed to update monitor: {error_msg}")
        except Exception as e:
            logger.error(f"Failed to update monitor {monitor_id}: {e}")
            raise
    async def delete_monitor(self, monitor_id: int) -> bool:
        """
        Delete a monitor
        Args:
            monitor_id: Monitor identifier
        Returns:
            True if successful
        """
        await self._ensure_connected()
        try:
            # Uptime Kuma delete monitor event
            response = await self.sio.call('deleteMonitor', monitor_id, timeout=self.timeout)
            if response and response.get('ok'):
                logger.info(f"Deleted monitor {monitor_id}")
                # Remove from cache
                self._monitors_cache.pop(monitor_id, None)
                return True
            error_msg = response.get('msg', 'Unknown error') if response else 'No response'
            raise Exception(f"Failed to delete monitor: {error_msg}")
        except Exception as e:
            logger.error(f"Failed to delete monitor {monitor_id}: {e}")
            raise
    async def delete_monitor_by_name(self, name: str) -> bool:
        """
        Delete a monitor by its name
        Args:
            name: Monitor name
        Returns:
            True if successful, False if monitor not found
        """
        monitor = await self.find_monitor_by_name(name)
        if not monitor:
            logger.warning(f"Monitor '{name}' not found")
            return False
        await self.delete_monitor(monitor["id"])
        return True
    async def __aenter__(self):
        """Async context manager entry"""
        await self._ensure_connected()
        return self
    async def __aexit__(self, exc_type, exc_val, exc_tb):
        """Async context manager exit"""
        await self.disconnect()
 # Singleton instance
 _kuma_client: Optional[KumaClient] = None
 def get_kuma_client() -> KumaClient:
    """Get singleton Kuma client instance"""
    global _kuma_client
    if _kuma_client is None:
        _kuma_client = KumaClient()
    return _kuma_client
@@ -99,9 +99,11 @@ class NPMClient:
            True if accessible, False otherwise
        """
        try:
-            async with httpx.AsyncClient(timeout=self.timeout) as client:
+            async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
                response = await client.get(f"{self.base_url}/api")
-                return response.status_code == 200
+                # Accept any successful response (2xx) or redirect (3xx) as healthy
                # A redirect indicates the service is up and responding
                return 200 <= response.status_code < 400
        except Exception as e:
            logger.error(f"NPM health check failed: {e}")
            return False
@@ -207,6 +209,116 @@ class NPMClient:
            response.raise_for_status()
            return response.json()
    async def update_proxy_host(
        self,
        proxy_id: int,
        config: Dict[str, Any]
    ) -> Dict[str, Any]:
        """
        Update an existing proxy host configuration
        Args:
            proxy_id: Proxy host ID to update
            config: Full proxy host configuration (get from get_proxy_host, modify, then update)
        Returns:
            Updated proxy host details
        """
        await self._ensure_token()
        async with httpx.AsyncClient(timeout=self.timeout) as client:
            response = await client.put(
                f"{self.base_url}/api/nginx/proxy-hosts/{proxy_id}",
                headers=self._get_headers(),
                json=config
            )
            if not response.is_success:
                logger.error(f"Update failed: {response.status_code}")
                logger.error(f"Response: {response.text}")
            response.raise_for_status()
            return response.json()
    async def enable_authentik_forward_auth(
        self,
        proxy_id: int,
        authentik_url: str = "http://authentik-server:9000"
    ) -> Dict[str, Any]:
        """
        Enable Authentik forward authentication on a proxy host
        Args:
            proxy_id: Proxy host ID to update
            authentik_url: Authentik server URL (default: http://authentik-server:9000)
        Returns:
            Updated proxy host details
        """
        # Get current config
        proxy_host = await self.get_proxy_host(proxy_id)
        # Authentik forward auth configuration
        auth_config = f"""# Authentik Forward Authentication
 # Send authentication requests to Authentik
 auth_request /outpost.goauthentik.io/auth/nginx;
 # Preserve authentication cookies
 auth_request_set $auth_cookie $upstream_http_set_cookie;
 add_header Set-Cookie $auth_cookie;
 # Get user information from Authentik
 auth_request_set $authentik_username $upstream_http_x_authentik_username;
 auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
 auth_request_set $authentik_email $upstream_http_x_authentik_email;
 auth_request_set $authentik_name $upstream_http_x_authentik_name;
 auth_request_set $authentik_uid $upstream_http_x_authentik_uid;
 # Pass user info to backend
 proxy_set_header X-authentik-username $authentik_username;
 proxy_set_header X-authentik-groups $authentik_groups;
 proxy_set_header X-authentik-email $authentik_email;
 proxy_set_header X-authentik-name $authentik_name;
 proxy_set_header X-authentik-uid $authentik_uid;
 # On authentication failure, redirect to Authentik login
 error_page 401 = @authentik_proxy_signin;
 location @authentik_proxy_signin {{
    internal;
    add_header Set-Cookie $auth_cookie;
    return 302 /outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
 }}
 # Authentik authentication endpoint
 location /outpost.goauthentik.io {{
    proxy_pass {authentik_url}/outpost.goauthentik.io;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header Host $host;
 }}
 """
        # Update the advanced config
        proxy_host["advanced_config"] = auth_config
        # Remove read-only fields that NPM doesn't accept in updates
        readonly_fields = [
            "id", "created_on", "modified_on", "owner", "owner_user_id",
            "certificate", "use_default_location", "ipv6", "meta", "nginx_online",
            "nginx_err", "access_list", "certificate_id"
        ]
        clean_config = {k: v for k, v in proxy_host.items() if k not in readonly_fields}
        # Ensure locations is an array (required field)
        if "locations" not in clean_config or clean_config["locations"] is None:
            clean_config["locations"] = []
        # Update the proxy host
        return await self.update_proxy_host(proxy_id, clean_config)
    async def get_certificates(self) -> List[Dict[str, Any]]:
        """
        List all SSL certificates
@@ -208,6 +208,87 @@ class PortainerClient:
            response.raise_for_status()
            return True
    async def get_containers(self, endpoint_id: int, all_containers: bool = True) -> List[Dict[str, Any]]:
        """
        List containers on a specific endpoint
        Args:
            endpoint_id: Portainer endpoint identifier
            all_containers: Include stopped containers (default: True)
        Returns:
            List of container details
        """
        params = {"all": 1 if all_containers else 0}
        async with httpx.AsyncClient(timeout=self.timeout) as client:
            response = await client.get(
                f"{self.base_url}/api/endpoints/{endpoint_id}/docker/containers/json",
                headers=self._get_headers(),
                params=params
            )
            response.raise_for_status()
            return response.json()
    async def get_container(self, endpoint_id: int, container_id: str) -> Dict[str, Any]:
        """
        Get detailed information about a specific container
        Args:
            endpoint_id: Portainer endpoint identifier
            container_id: Container ID or name
        Returns:
            Container details including network and port information
        """
        async with httpx.AsyncClient(timeout=self.timeout) as client:
            response = await client.get(
                f"{self.base_url}/api/endpoints/{endpoint_id}/docker/containers/{container_id}/json",
                headers=self._get_headers()
            )
            response.raise_for_status()
            return response.json()
    async def stop_container(self, endpoint_id: int, container_id: str) -> bool:
        """
        Stop a container
        Args:
            endpoint_id: Portainer endpoint identifier
            container_id: Container ID or name
        Returns:
            True if successful
        """
        async with httpx.AsyncClient(timeout=self.timeout) as client:
            response = await client.post(
                f"{self.base_url}/api/endpoints/{endpoint_id}/docker/containers/{container_id}/stop",
                headers=self._get_headers()
            )
            response.raise_for_status()
            logger.info(f"Stopped container {container_id}")
            return True
    async def start_container(self, endpoint_id: int, container_id: str) -> bool:
        """
        Start a container
        Args:
            endpoint_id: Portainer endpoint identifier
            container_id: Container ID or name
        Returns:
            True if successful
        """
        async with httpx.AsyncClient(timeout=self.timeout) as client:
            response = await client.post(
                f"{self.base_url}/api/endpoints/{endpoint_id}/docker/containers/{container_id}/start",
                headers=self._get_headers()
            )
            response.raise_for_status()
            logger.info(f"Started container {container_id}")
            return True
 # Singleton instance
 _portainer_client: Optional[PortainerClient] = None
@@ -89,6 +89,11 @@ class Settings(BaseSettings):
    kuma_username: str = KUMA_USERNAME
    kuma_password: str = KUMA_PASSWORD
    # OIDC Authentication (Authentik)
    oidc_enabled: bool = False  # Set to True to require authentication
    oidc_issuer: str = "https://auth.schweitz.net/application/o/core-api/"
    oidc_audience: str = "core-api"
    @property
    def model_aliases(self) -> dict:
        """Computed property for model aliases"""
@@ -0,0 +1,671 @@
 """
 AI Controller
 Provides AI orchestration endpoints including:
 - OpenAI-compatible chat completions
 - Model listing
 - Conversation memory management
 """
 import time
 import logging
 import uuid
 from fastapi import APIRouter, HTTPException, Query
 from fastapi.responses import StreamingResponse
 from typing import AsyncIterator, List, Optional
 from pydantic import BaseModel, Field
 from src.controllers.base import BaseController
 from src.api.v1.schemas import (
    ChatCompletionRequest,
    ChatCompletionResponse,
    ChatCompletionChoice,
    ChatMessageResponse,
    UsageInfo,
    ChatCompletionStreamResponse,
    ChatCompletionStreamChoice,
    DeltaMessage,
    ModelsListResponse,
    ModelInfo,
 )
 from src.models.ollama_client import get_ollama_client
 from src.memory import get_memory_manager, MessageRole as MemoryMessageRole, TokenUsage
 from src.config import get_settings
 logger = logging.getLogger(__name__)
 # Request/Response Models for Conversations
 class SearchRequest(BaseModel):
    """Request model for semantic search"""
    query: str = Field(..., description="Search query")
    limit: int = Field(5, ge=1, le=50, description="Maximum number of results")
 class ConversationTurnResponse(BaseModel):
    """Response model for a conversation turn"""
    turn_number: int
    role: str
    content: str
    timestamp: str
    tokens_prompt: Optional[int] = None
    tokens_completion: Optional[int] = None
    tokens_total: Optional[int] = None
    metadata: dict = Field(default_factory=dict)
 class ConversationHistoryResponse(BaseModel):
    """Response model for conversation history"""
    conversation_id: str
    turn_count: int
    total_tokens: int
    turns: List[ConversationTurnResponse]
 class SearchResultResponse(BaseModel):
    """Response model for a single search result"""
    conversation_id: str
    turn_number: int
    role: str
    content: str
    timestamp: str
    score: float
 class SearchResponse(BaseModel):
    """Response model for search results"""
    query: str
    results: List[SearchResultResponse]
    count: int
 class ConversationStatsResponse(BaseModel):
    """Response model for conversation statistics"""
    conversation_id: str
    buffer_turns: int
    buffer_tokens: int
    qdrant_turns: int
    qdrant_tokens: int
    exists_in_buffer: bool
    exists_in_qdrant: bool
 class DeleteResponse(BaseModel):
    """Response model for delete operation"""
    conversation_id: str
    deleted: bool
    message: str
 # Helper functions
 def build_prompt_from_messages(messages: list) -> str:
    """
    Convert message list to a prompt string.
    """
    prompt_parts = []
    for msg in messages:
        role = msg.role.value if hasattr(msg.role, 'value') else msg.role
        content = msg.content
        if role == "system":
            prompt_parts.append(f"System: {content}")
        elif role == "user":
            prompt_parts.append(f"User: {content}")
        elif role == "assistant":
            prompt_parts.append(f"Assistant: {content}")
    prompt_parts.append("Assistant:")
    return "\n\n".join(prompt_parts)
 async def stream_chat_completion(
    request_id: str,
    model: str,
    prompt: str,
    temperature: float,
    max_tokens: int | None
 ) -> AsyncIterator[str]:
    """
    Stream chat completion in OpenAI SSE format.
    Yields:
        Server-Sent Events formatted strings
    """
    created = int(time.time())
    ollama_client = get_ollama_client()
    # First chunk with role
    first_chunk = ChatCompletionStreamResponse(
        id=request_id,
        created=created,
        model=model,
        choices=[
            ChatCompletionStreamChoice(
                index=0,
                delta=DeltaMessage(role="assistant"),
                finish_reason=None
            )
        ]
    )
    yield f"data: {first_chunk.model_dump_json()}\n\n"
    # Stream tokens
    try:
        async for token in ollama_client.generate_streaming(
            model=model,
            prompt=prompt,
            temperature=temperature,
            max_tokens=max_tokens
        ):
            chunk = ChatCompletionStreamResponse(
                id=request_id,
                created=created,
                model=model,
                choices=[
                    ChatCompletionStreamChoice(
                        index=0,
                        delta=DeltaMessage(content=token),
                        finish_reason=None
                    )
                ]
            )
            yield f"data: {chunk.model_dump_json()}\n\n"
    except Exception as e:
        logger.error(f"Streaming error: {e}")
        # Send error in OpenAI format
        import json
        error_chunk = {
            "error": {
                "message": str(e),
                "type": "server_error"
            }
        }
        yield f"data: {json.dumps(error_chunk)}\n\n"
        return
    # Final chunk
    final_chunk = ChatCompletionStreamResponse(
        id=request_id,
        created=created,
        model=model,
        choices=[
            ChatCompletionStreamChoice(
                index=0,
                delta=DeltaMessage(),
                finish_reason="stop"
            )
        ]
    )
    yield f"data: {final_chunk.model_dump_json()}\n\n"
    yield "data: [DONE]\n\n"
 async def store_conversation_turn(
    conversation_id: str,
    role: str,
    content: str,
    tokens: dict = None
 ):
    """
    Store a conversation turn in memory
    Args:
        conversation_id: Unique conversation identifier
        role: Message role (user, assistant, system)
        content: Message content
        tokens: Optional token usage dict
    """
    try:
        memory_manager = get_memory_manager()
        # Convert role string to MemoryMessageRole
        if role == "user":
            memory_role = MemoryMessageRole.USER
        elif role == "assistant":
            memory_role = MemoryMessageRole.ASSISTANT
        elif role == "system":
            memory_role = MemoryMessageRole.SYSTEM
        else:
            memory_role = MemoryMessageRole.USER  # Default fallback
        # Create TokenUsage if provided
        token_usage = None
        if tokens:
            token_usage = TokenUsage(
                prompt=tokens.get("prompt", 0),
                completion=tokens.get("completion", 0),
                total=tokens.get("total", 0)
            )
        # Store in memory
        await memory_manager.add_turn(
            conversation_id=conversation_id,
            role=memory_role,
            content=content,
            tokens=token_usage
        )
        logger.debug(f"Stored {role} turn in memory for conversation {conversation_id}")
    except Exception as e:
        # Log error but don't fail the request
        logger.error(f"Failed to store turn in memory: {e}")
 class AIController(BaseController):
    """
    Controller for AI orchestration operations
    Provides endpoints for:
    - OpenAI-compatible chat completions (streaming and non-streaming)
    - Model listing
    - Conversation memory management
    """
    def __init__(self):
        super().__init__(prefix="/v1", tags=["AI"])
    def create_router(self) -> APIRouter:
        """Create and configure the router"""
        router = APIRouter()
        settings = get_settings()
        # Chat Completions Endpoint
        @router.post(
            "/v1/chat/completions",
            tags=["AI"]
        )
        async def chat_completions(request: ChatCompletionRequest):
            """
            OpenAI-compatible chat completions endpoint.
            Supports both streaming and non-streaming.
            Automatically stores conversations in memory system if enabled.
            """
            request_id = f"chatcmpl-{int(time.time() * 1000)}"
            # Generate or use provided conversation_id
            conversation_id = request.conversation_id or f"conv_{uuid.uuid4().hex[:16]}"
            logger.info(
                f"Chat request: id={request_id}, model={request.model}, "
                f"messages={len(request.messages)}, stream={request.stream}, "
                f"conversation_id={conversation_id}, store_in_memory={request.store_in_memory}"
            )
            # Store user messages in memory (if enabled)
            if request.store_in_memory:
                for msg in request.messages:
                    role = msg.role.value if hasattr(msg.role, 'value') else msg.role
                    if role == "user":  # Store latest user message
                        await store_conversation_turn(
                            conversation_id=conversation_id,
                            role=role,
                            content=msg.content
                        )
            # Build prompt from messages
            prompt = build_prompt_from_messages(request.messages)
            # Streaming response
            if request.stream:
                return StreamingResponse(
                    stream_chat_completion(
                        request_id=request_id,
                        model=request.model,
                        prompt=prompt,
                        temperature=request.temperature,
                        max_tokens=request.max_tokens
                    ),
                    media_type="text/event-stream"
                )
            # Non-streaming response
            try:
                ollama_client = get_ollama_client()
                result = await ollama_client.generate_non_streaming(
                    model=request.model,
                    prompt=prompt,
                    temperature=request.temperature,
                    max_tokens=request.max_tokens
                )
                assistant_content = result["response"]
                # Store assistant response in memory (if enabled)
                if request.store_in_memory:
                    await store_conversation_turn(
                        conversation_id=conversation_id,
                        role="assistant",
                        content=assistant_content,
                        tokens=result["tokens"]
                    )
                response = ChatCompletionResponse(
                    id=request_id,
                    created=int(time.time()),
                    model=request.model,
                    choices=[
                        ChatCompletionChoice(
                            index=0,
                            message=ChatMessageResponse(
                                role="assistant",
                                content=assistant_content
                            ),
                            finish_reason="stop"
                        )
                    ],
                    usage=UsageInfo(
                        prompt_tokens=result["tokens"]["prompt"],
                        completion_tokens=result["tokens"]["completion"],
                        total_tokens=result["tokens"]["total"]
                    )
                )
                logger.info(
                    f"Chat response: id={request_id}, "
                    f"tokens={result['tokens']['total']}, "
                    f"conversation_id={conversation_id}"
                )
                return response
            except Exception as e:
                logger.error(f"Chat completion error: {e}")
                raise HTTPException(
                    status_code=500,
                    detail=f"Failed to generate completion: {str(e)}"
                )
        # Models Endpoint
        @router.get(
            "/v1/models",
            tags=["AI"]
        )
        async def list_models():
            """List available models in OpenAI format."""
            models = []
            # Add OpenAI-style aliases
            for alias in settings.model_aliases.keys():
                models.append(ModelInfo(id=alias, owned_by="tatlock"))
            # Add actual local models
            for model_list in [
                settings.get_lightweight_models(),
                settings.get_heavy_models(),
                settings.get_code_models()
            ]:
                for model in model_list:
                    # Avoid duplicates
                    if model not in [m.id for m in models]:
                        models.append(ModelInfo(id=model, owned_by="tatlock"))
            return ModelsListResponse(data=models)
        # Conversation Endpoints
        @router.get(
            "/v1/conversations/{conversation_id}",
            response_model=ConversationHistoryResponse,
            tags=["Conversations"],
            summary="Get conversation history",
            description="Retrieve complete conversation history including all turns"
        )
        async def get_conversation(
            conversation_id: str,
            include_buffer: bool = Query(
                True,
                description="Include recent turns from buffer that haven't been consolidated yet"
            )
        ):
            """
            Get complete conversation history
            Args:
                conversation_id: Unique conversation identifier
                include_buffer: Include recent buffer turns not yet consolidated
            Returns:
                Complete conversation history with all turns
            """
            manager = get_memory_manager()
            # Get full history
            turns = await manager.get_full_history(conversation_id, include_buffer=include_buffer)
            if not turns:
                raise HTTPException(
                    status_code=404,
                    detail=f"Conversation {conversation_id} not found"
                )
            # Convert to response format
            turn_responses = []
            total_tokens = 0
            for turn in turns:
                turn_response = ConversationTurnResponse(
                    turn_number=turn.turn_number,
                    role=turn.role.value if isinstance(turn.role, MemoryMessageRole) else turn.role,
                    content=turn.content,
                    timestamp=turn.timestamp.isoformat(),
                    metadata=turn.metadata
                )
                if turn.tokens:
                    turn_response.tokens_prompt = turn.tokens.prompt
                    turn_response.tokens_completion = turn.tokens.completion
                    turn_response.tokens_total = turn.tokens.total
                    total_tokens += turn.tokens.total
                turn_responses.append(turn_response)
            return ConversationHistoryResponse(
                conversation_id=conversation_id,
                turn_count=len(turns),
                total_tokens=total_tokens,
                turns=turn_responses
            )
        @router.get(
            "/v1/conversations/{conversation_id}/stats",
            response_model=ConversationStatsResponse,
            tags=["Conversations"],
            summary="Get conversation statistics",
            description="Get detailed statistics about a conversation across all storage tiers"
        )
        async def get_conversation_stats(conversation_id: str):
            """
            Get conversation statistics
            Args:
                conversation_id: Unique conversation identifier
            Returns:
                Statistics including turn counts and token usage across tiers
            """
            manager = get_memory_manager()
            stats = await manager.get_conversation_stats(conversation_id)
            return ConversationStatsResponse(**stats)
        @router.post(
            "/v1/conversations/{conversation_id}/search",
            response_model=SearchResponse,
            tags=["Conversations"],
            summary="Search conversation semantically",
            description="Search for relevant turns within a conversation using semantic similarity"
        )
        async def search_conversation(
            conversation_id: str,
            search_request: SearchRequest
        ):
            """
            Semantic search within a conversation
            Args:
                conversation_id: Unique conversation identifier
                search_request: Search query and parameters
            Returns:
                Relevant conversation turns ranked by semantic similarity
            """
            manager = get_memory_manager()
            # Perform semantic search
            results = await manager.search_conversations(
                query=search_request.query,
                conversation_id=conversation_id,
                limit=search_request.limit
            )
            # Convert to response format
            search_results = [
                SearchResultResponse(
                    conversation_id=result["conversation_id"],
                    turn_number=result["turn_number"],
                    role=result["role"],
                    content=result["content"],
                    timestamp=result["timestamp"],
                    score=result["score"]
                )
                for result in results
            ]
            return SearchResponse(
                query=search_request.query,
                results=search_results,
                count=len(search_results)
            )
        @router.post(
            "/v1/conversations/search",
            response_model=SearchResponse,
            tags=["Conversations"],
            summary="Search all conversations",
            description="Search across all conversations using semantic similarity"
        )
        async def search_all_conversations(search_request: SearchRequest):
            """
            Semantic search across all conversations
            Args:
                search_request: Search query and parameters
            Returns:
                Relevant turns from any conversation ranked by semantic similarity
            """
            manager = get_memory_manager()
            # Perform semantic search across all conversations
            results = await manager.search_conversations(
                query=search_request.query,
                conversation_id=None,  # Search all conversations
                limit=search_request.limit
            )
            # Convert to response format
            search_results = [
                SearchResultResponse(
                    conversation_id=result["conversation_id"],
                    turn_number=result["turn_number"],
                    role=result["role"],
                    content=result["content"],
                    timestamp=result["timestamp"],
                    score=result["score"]
                )
                for result in results
            ]
            return SearchResponse(
                query=search_request.query,
                results=search_results,
                count=len(search_results)
            )
        @router.delete(
            "/v1/conversations/{conversation_id}",
            response_model=DeleteResponse,
            tags=["Conversations"],
            summary="Delete conversation",
            description="Delete a conversation from all storage tiers"
        )
        async def delete_conversation(
            conversation_id: str,
            clear_buffer: bool = Query(True, description="Clear from buffer (Tier 1)"),
            clear_qdrant: bool = Query(True, description="Clear from Qdrant (Tier 2/3)")
        ):
            """
            Delete a conversation
            Args:
                conversation_id: Unique conversation identifier
                clear_buffer: Clear from Tier 1 buffer
                clear_qdrant: Clear from Tier 2/3 Qdrant
            Returns:
                Deletion confirmation
            """
            manager = get_memory_manager()
            try:
                await manager.clear_conversation(
                    conversation_id,
                    clear_buffer=clear_buffer,
                    clear_qdrant=clear_qdrant
                )
                return DeleteResponse(
                    conversation_id=conversation_id,
                    deleted=True,
                    message=f"Conversation {conversation_id} deleted successfully"
                )
            except Exception as e:
                raise HTTPException(
                    status_code=500,
                    detail=f"Error deleting conversation: {str(e)}"
                )
        @router.post(
            "/v1/conversations/{conversation_id}/consolidate",
            tags=["Conversations"],
            summary="Consolidate conversation",
            description="Manually trigger consolidation from buffer to persistent storage"
        )
        async def consolidate_conversation(conversation_id: str):
            """
            Manually consolidate a conversation
            Moves all buffer turns to Qdrant persistent storage.
            Args:
                conversation_id: Unique conversation identifier
            Returns:
                Number of turns consolidated
            """
            manager = get_memory_manager()
            try:
                count = await manager.consolidate(conversation_id)
                return {
                    "conversation_id": conversation_id,
                    "consolidated_turns": count,
                    "message": f"Successfully consolidated {count} turns to persistent storage"
                }
            except Exception as e:
                raise HTTPException(
                    status_code=500,
                    detail=f"Error consolidating conversation: {str(e)}"
                )
        return router
 # Create controller instance
 ai_controller = AIController()
@@ -0,0 +1,89 @@
 """
 Health Controller
 Provides service health and information endpoints
 """
 from fastapi import APIRouter
 from fastapi.responses import JSONResponse
 from src.controllers.base import BaseController
 from src.config import get_settings
 from src.logging_config import get_logger
 from src.models.ollama_client import get_ollama_client
 logger = get_logger(__name__)
 class HealthController(BaseController):
    """
    Controller for service health and information
    Provides endpoints for:
    - Service information and status
    - Health checks
    """
    def __init__(self):
        super().__init__(prefix="", tags=["Health"])
    def create_router(self) -> APIRouter:
        """Create and configure the router"""
        router = APIRouter(tags=self.tags)
        settings = get_settings()
        @router.get(
            "/",
            summary="Service information",
            response_class=JSONResponse
        )
        async def root():
            """
            Get service information and health status
            Returns basic information about the API service and available endpoints.
            """
            logger.debug("Root endpoint accessed")
            return {
                "service": settings.app_name,
                "version": settings.app_version,
                "status": "healthy",
                "documentation": {
                    "swagger_ui": "/docs",
                    "redoc": "/redoc",
                    "openapi_spec": "/openapi.json"
                },
                "endpoints": {
                    "chat_completions": "/v1/chat/completions",
                    "models": "/v1/models",
                    "conversations": "/v1/conversations",
                    "web_scraper": "/web-scraper/scrape",
                    "infrastructure": "/infrastructure",
                    "health": "/health"
                }
            }
        @router.get(
            "/health",
            summary="Health check",
            response_class=JSONResponse
        )
        async def health_check():
            """
            Simple health check endpoint for container orchestration
            Returns a 200 OK status when the service is running properly.
            Used by Docker, Kubernetes, and load balancers.
            """
            ollama_client = get_ollama_client()
            ollama_healthy = await ollama_client.health_check()
            return {
                "status": "healthy",
                "ollama_connected": ollama_healthy
            }
        return router
 # Create controller instance
 health_controller = HealthController()
@@ -4,14 +4,17 @@ Infrastructure Management Controller
 Provides API endpoints for automated infrastructure management,
 including service deployment, configuration, and monitoring setup.
 """
-from fastapi import APIRouter, HTTPException
+from fastapi import APIRouter, HTTPException, Depends
 from typing import List, Dict, Any, Optional, Union
 from pydantic import BaseModel, field_validator
 from src.controllers.base import BaseController
 from src.clients.portainer_client import get_portainer_client
 from src.clients.npm_client import get_npm_client
 from src.clients.kuma_client import get_kuma_client
 from src.logging_config import get_logger
 from src import service_groups
 from src.auth.oidc import get_admin_user
 logger = get_logger(__name__)
@@ -25,6 +28,9 @@ class ServiceInfo(BaseModel):
    endpoint_id: Optional[int]
    ports: List[int] = []
    domains: List[str] = []
    running: bool = False  # True if at least one container is running
    containers_running: int = 0  # Number of running containers
    containers_total: int = 0  # Total number of containers
    @field_validator('status', mode='before')
    @classmethod
@@ -40,7 +46,12 @@ class PortInfo(BaseModel):
    """Information about an allocated port"""
    port: int
    service: str
    container_name: Optional[str] = None
    protocol: str = "tcp"
    internal_hostname: Optional[str] = None
    internal_ip: Optional[str] = None
    external_domains: List[str] = []
    host_port: Optional[int] = None  # Port exposed on host, if different from internal
    description: str = ""
@@ -158,17 +169,19 @@ class InfrastructureController(BaseController):
        @router.get(
            "/services",
            response_model=List[ServiceInfo],
-            summary="List all deployed services"
+            summary="List all deployed services (Docker Compose stacks)",
            description="List all services deployed via Portainer stacks. Each 'service' represents a Docker Compose stack."
        )
        async def list_services():
            """
-            List all deployed services from Portainer stacks
+            List all deployed Docker Compose stacks from Portainer
-            Returns comprehensive service information including:
+            Returns comprehensive stack/service information including:
-            - Stack/service name
+            - Stack name (the service name)
-            - Status
+            - Status (active/inactive)
            - Exposed ports
-            - Configured domains
+            - Configured domains (from NPM reverse proxy)
            - Running container count
            """
            portainer = get_portainer_client()
            npm = get_npm_client()
@@ -189,18 +202,38 @@ class InfrastructureController(BaseController):
                for stack in stacks:
                    # Find domains for this stack
                    stack_name = stack.get("Name", "")
                    endpoint_id = stack.get("EndpointId")
                    domains = [
                        domain for domain, host in domain_map.items()
                        if stack_name in host or host in stack_name
                    ]
                    # Get container status for this stack
                    containers_running = 0
                    containers_total = 0
                    try:
                        all_containers = await portainer.get_containers(endpoint_id, all_containers=True)
                        for container in all_containers:
                            labels = container.get("Labels", {})
                            container_stack = labels.get("com.docker.compose.project", "")
                            if container_stack.lower() == stack_name.lower():
                                containers_total += 1
                                if container.get("State", "") == "running":
                                    containers_running += 1
                    except Exception as e:
                        logger.warning(f"Failed to get container status for {stack_name}: {e}")
                    service_info = ServiceInfo(
                        name=stack_name,
                        stack_id=stack.get("Id"),
                        status=stack.get("Status", "unknown"),
-                        endpoint_id=stack.get("EndpointId"),
+                        endpoint_id=endpoint_id,
                        ports=[],  # TODO: Extract from stack file
-                        domains=domains
+                        domains=domains,
                        running=containers_running > 0,
                        containers_running=containers_running,
                        containers_total=containers_total
                    )
                    services.append(service_info)
@@ -263,12 +296,133 @@ class InfrastructureController(BaseController):
            """
            List all currently allocated ports
-            Scans services and proxy configurations to build
+            Scans all running containers to extract:
-            a comprehensive port allocation map.
+            - Internal and external port mappings
            - Container internal hostnames and IPs
            - External domain names (from NPM proxy configuration)
            """
-            # TODO: Implement port scanning from containers and proxy configs
+            portainer = get_portainer_client()
-            # For now, return a placeholder
+            npm = get_npm_client()
-            return []
+
            try:
                # Get all endpoints (Docker environments)
                endpoints = await portainer.get_endpoints()
                # Get proxy hosts for external domain mapping
                proxy_hosts = await npm.get_proxy_hosts()
                # Build mapping of forward_host:forward_port -> domains
                port_domain_map = {}
                for proxy in proxy_hosts:
                    forward_host = proxy.get("forward_host", "")
                    forward_port = proxy.get("forward_port", 0)
                    domains = proxy.get("domain_names", [])
                    key = f"{forward_host}:{forward_port}"
                    if key not in port_domain_map:
                        port_domain_map[key] = []
                    port_domain_map[key].extend(domains)
                ports = []
                # Scan containers on each endpoint
                for endpoint in endpoints:
                    endpoint_id = endpoint.get("Id")
                    try:
                        containers = await portainer.get_containers(endpoint_id, all_containers=False)
                        for container in containers:
                            container_name = container.get("Names", ["unknown"])[0].lstrip("/")
                            state = container.get("State", "")
                            # Skip non-running containers
                            if state != "running":
                                continue
                            # Extract network information
                            networks = container.get("NetworkSettings", {}).get("Networks", {})
                            internal_hostname = container_name
                            internal_ip = None
                            # Get first network IP
                            for network_name, network_info in networks.items():
                                if network_info.get("IPAddress"):
                                    internal_ip = network_info.get("IPAddress")
                                    break
                            # Extract port mappings
                            port_mappings = container.get("Ports", [])
                            for port_mapping in port_mappings:
                                internal_port = port_mapping.get("PrivatePort")
                                host_port = port_mapping.get("PublicPort")
                                protocol = port_mapping.get("Type", "tcp")
                                if not internal_port:
                                    continue
                                # Find external domains for this port
                                external_domains = []
                                # Try matching by container name and port
                                key_by_name = f"{container_name}:{internal_port}"
                                if key_by_name in port_domain_map:
                                    external_domains.extend(port_domain_map[key_by_name])
                                # Try matching by internal IP and port
                                if internal_ip:
                                    key_by_ip = f"{internal_ip}:{internal_port}"
                                    if key_by_ip in port_domain_map:
                                        external_domains.extend(port_domain_map[key_by_ip])
                                # Try matching by localhost and host port
                                if host_port:
                                    for localhost_variant in ["localhost", "127.0.0.1", "192.168.86.149"]:
                                        key_by_host = f"{localhost_variant}:{host_port}"
                                        if key_by_host in port_domain_map:
                                            external_domains.extend(port_domain_map[key_by_host])
                                # Deduplicate external domains
                                external_domains = list(set(external_domains))
                                # Get service name from stack label or container name
                                labels = container.get("Labels", {})
                                service_name = labels.get("com.docker.compose.service", container_name)
                                port_info = PortInfo(
                                    port=internal_port,
                                    service=service_name,
                                    container_name=container_name,
                                    protocol=protocol,
                                    internal_hostname=internal_hostname,
                                    internal_ip=internal_ip,
                                    external_domains=external_domains,
                                    host_port=host_port,
                                    description=f"{container_name} on {endpoint.get('Name', 'unknown')}"
                                )
                                ports.append(port_info)
                    except Exception as e:
                        logger.error(f"Failed to scan containers on endpoint {endpoint_id}: {e}")
                        continue
                # Deduplicate ports based on (port, container_name, protocol)
                seen = set()
                unique_ports = []
                for port_info in ports:
                    key = (port_info.port, port_info.container_name, port_info.protocol)
                    if key not in seen:
                        seen.add(key)
                        unique_ports.append(port_info)
                # Sort by port number
                unique_ports.sort(key=lambda p: p.port)
                return unique_ports
            except Exception as e:
                logger.error(f"Failed to list ports: {e}")
                raise HTTPException(status_code=500, detail=str(e))
        @router.get(
            "/domains",
@@ -311,18 +465,32 @@ class InfrastructureController(BaseController):
        @router.post(
            "/services",
            response_model=OperationResult,
-            summary="Deploy a new service",
+            summary="Deploy a new Docker Compose stack",
            description="Deploy a new service by creating a Docker Compose stack in Portainer. Provide the stack name and compose file content. Requires admin authentication.",
            status_code=201
        )
-        async def deploy_service(request: DeployServiceRequest):
+        async def deploy_service(
            request: DeployServiceRequest,
            user: Dict = Depends(get_admin_user)
        ):
            """
-            Deploy a new service via Portainer stack
+            Deploy a new Docker Compose stack via Portainer
            Creates a new Portainer stack from the provided Docker Compose content.
            This is equivalent to deploying a stack through the Portainer UI.
            Args:
-                request: Service deployment configuration
+                request: Stack deployment configuration containing:
                    - name: Stack name (must be unique)
                    - compose_content: Full docker-compose.yml content as string
                    - endpoint_id: Portainer endpoint ID (default: 3 for local)
            Returns:
-                Operation result with stack details
+                Operation result with created stack details including stack ID
            Raises:
                409: If a stack with the same name already exists
                500: If deployment fails
            """
            portainer = get_portainer_client()
@@ -360,11 +528,19 @@ class InfrastructureController(BaseController):
        @router.put(
            "/services/{name}",
            response_model=OperationResult,
-            summary="Update an existing service"
+            summary="Update an existing Docker Compose stack",
            description="Update a deployed stack's Docker Compose configuration. This redeploys the stack with the new configuration. Requires admin authentication."
        )
-        async def update_service(name: str, request: UpdateServiceRequest):
+        async def update_service(
            name: str,
            request: UpdateServiceRequest,
            user: Dict = Depends(get_admin_user)
        ):
            """
-            Update an existing service's compose configuration
+            Update an existing Docker Compose stack's configuration
            Updates the stack's compose file and redeploys it. This is equivalent
            to updating a stack through the Portainer UI.
            Args:
                name: Service/stack name
@@ -415,9 +591,13 @@ class InfrastructureController(BaseController):
        @router.delete(
            "/services/{name}",
            response_model=OperationResult,
-            summary="Delete a service"
+            summary="Delete a service",
            description="Delete a service and remove its stack. Requires admin authentication."
        )
-        async def delete_service(name: str):
+        async def delete_service(
            name: str,
            user: Dict = Depends(get_admin_user)
        ):
            """
            Delete a service and remove its stack
@@ -463,13 +643,40 @@ class InfrastructureController(BaseController):
                logger.error(f"Failed to delete service '{name}': {e}")
                raise HTTPException(status_code=500, detail=str(e))
        @router.get(
            "/proxy/{proxy_id}",
            summary="Get proxy host details"
        )
        async def get_proxy_host(proxy_id: int):
            """
            Get detailed configuration of a specific proxy host
            Args:
                proxy_id: NPM proxy host ID
            Returns:
                Complete proxy host configuration including locations
            """
            npm = get_npm_client()
            try:
                proxy_host = await npm.get_proxy_host(proxy_id)
                return proxy_host
            except Exception as e:
                logger.error(f"Failed to get proxy host {proxy_id}: {e}")
                raise HTTPException(status_code=500, detail=str(e))
        @router.post(
            "/proxy",
            response_model=OperationResult,
            summary="Create a new proxy host",
            description="Create a new Nginx Proxy Manager proxy host with optional SSL certificate. Requires admin authentication.",
            status_code=201
        )
-        async def create_proxy(request: CreateProxyRequest):
+        async def create_proxy(
            request: CreateProxyRequest,
            user: Dict = Depends(get_admin_user)
        ):
            """
            Create a new Nginx Proxy Manager proxy host
@@ -523,6 +730,410 @@ class InfrastructureController(BaseController):
                logger.error(f"Failed to create proxy host: {e}")
                raise HTTPException(status_code=500, detail=str(e))
        # Service Control Endpoints
        @router.get(
            "/service-groups",
            summary="List service groups"
        )
        async def list_service_groups():
            """
            List all defined service groups
            Returns service groups with their member services and status.
            """
            return {
                "groups": service_groups.list_service_groups(),
                "always_on": list(service_groups.ALWAYS_ON_SERVICES),
                "stoppable": service_groups.list_stoppable_services()
            }
        @router.post(
            "/services/{name}/stop",
            response_model=OperationResult,
            summary="Stop a service or service group",
            description="Stop a service or service group by pausing monitors and stopping containers. Requires admin authentication."
        )
        async def stop_service(
            name: str,
            user: Dict = Depends(get_admin_user)
        ):
            """
            Stop a service or service group
            This will:
            1. Validate service can be stopped (not always-on)
            2. Pause Uptime Kuma monitors for all services in group
            3. Stop the Portainer stack(s)
            Args:
                name: Service or group name
            Returns:
                Operation result with details
            """
            portainer = get_portainer_client()
            kuma = get_kuma_client()
            try:
                # Get all services in the group
                services = service_groups.get_service_group(name)
                # Validate none are always-on
                is_valid, error_msg = service_groups.validate_stop_request(services)
                if not is_valid:
                    raise HTTPException(status_code=403, detail=error_msg)
                results = {
                    "stopped_services": [],
                    "paused_monitors": [],
                    "errors": []
                }
                # Stop each service
                for service_name in services:
                    try:
                        # 1. Pause Uptime Kuma monitor
                        try:
                            monitor_paused = await kuma.pause_monitor_by_name(service_name)
                            if monitor_paused:
                                results["paused_monitors"].append(service_name)
                                logger.info(f"Paused Kuma monitor for {service_name}")
                        except Exception as e:
                            logger.warning(f"Failed to pause Kuma monitor for {service_name}: {e}")
                            results["errors"].append(f"Kuma pause failed for {service_name}: {str(e)}")
                        # 2. Stop Portainer stack
                        stacks = await portainer.get_stacks()
                        stack = next(
                            (s for s in stacks if s.get("Name", "").lower() == service_name.lower()),
                            None
                        )
                        if stack:
                            stack_id = stack.get("Id")
                            endpoint_id = stack.get("EndpointId")
                            # Stop stack by deleting it (Portainer doesn't have a "stop" operation)
                            # Note: This is destructive. For a gentler approach, we'd need to use docker compose stop
                            # Let's use docker API instead
                            logger.info(f"Stopping containers for stack: {service_name}")
                            # Get containers for this stack
                            containers = await portainer.get_containers(endpoint_id, all_containers=False)
                            stopped_containers = []
                            for container in containers:
                                labels = container.get("Labels", {})
                                container_stack = labels.get("com.docker.compose.project", "")
                                if container_stack.lower() == service_name.lower():
                                    container_id = container.get("Id")
                                    # Stop container via Portainer Docker API
                                    await portainer.stop_container(endpoint_id, container_id)
                                    stopped_containers.append(container.get("Names", ["unknown"])[0])
                            results["stopped_services"].append({
                                "service": service_name,
                                "stack_id": stack_id,
                                "containers": stopped_containers
                            })
                            logger.info(f"Stopped service: {service_name}")
                        else:
                            results["errors"].append(f"Stack not found: {service_name}")
                    except Exception as e:
                        logger.error(f"Failed to stop service {service_name}: {e}")
                        results["errors"].append(f"{service_name}: {str(e)}")
                success = len(results["stopped_services"]) > 0
                message = f"Stopped {len(results['stopped_services'])} service(s)"
                if results["errors"]:
                    message += f" with {len(results['errors'])} error(s)"
                return OperationResult(
                    success=success,
                    message=message,
                    details=results
                )
            except HTTPException:
                raise
            except Exception as e:
                logger.error(f"Failed to stop service group '{name}': {e}")
                raise HTTPException(status_code=500, detail=str(e))
        @router.post(
            "/services/{name}/start",
            response_model=OperationResult,
            summary="Start a service or service group",
            description="Start a service or service group by starting containers and resuming monitors. Requires admin authentication."
        )
        async def start_service(
            name: str,
            user: Dict = Depends(get_admin_user)
        ):
            """
            Start a service or service group
            This will:
            1. Start the Portainer stack(s)
            2. Resume Uptime Kuma monitors for all services in group
            Args:
                name: Service or group name
            Returns:
                Operation result with details
            """
            portainer = get_portainer_client()
            kuma = get_kuma_client()
            try:
                # Get all services in the group
                services = service_groups.get_service_group(name)
                results = {
                    "started_services": [],
                    "resumed_monitors": [],
                    "errors": []
                }
                # Start each service
                for service_name in services:
                    try:
                        # 1. Start Portainer stack (start containers)
                        stacks = await portainer.get_stacks()
                        stack = next(
                            (s for s in stacks if s.get("Name", "").lower() == service_name.lower()),
                            None
                        )
                        if stack:
                            stack_id = stack.get("Id")
                            endpoint_id = stack.get("EndpointId")
                            logger.info(f"Starting containers for stack: {service_name}")
                            # Get containers for this stack
                            containers = await portainer.get_containers(endpoint_id, all_containers=True)
                            started_containers = []
                            for container in containers:
                                labels = container.get("Labels", {})
                                container_stack = labels.get("com.docker.compose.project", "")
                                if container_stack.lower() == service_name.lower():
                                    container_id = container.get("Id")
                                    # Start container via Portainer Docker API
                                    await portainer.start_container(endpoint_id, container_id)
                                    started_containers.append(container.get("Names", ["unknown"])[0])
                            results["started_services"].append({
                                "service": service_name,
                                "stack_id": stack_id,
                                "containers": started_containers
                            })
                            logger.info(f"Started service: {service_name}")
                            # 2. Resume Uptime Kuma monitor
                            try:
                                monitor_resumed = await kuma.resume_monitor_by_name(service_name)
                                if monitor_resumed:
                                    results["resumed_monitors"].append(service_name)
                                    logger.info(f"Resumed Kuma monitor for {service_name}")
                            except Exception as e:
                                logger.warning(f"Failed to resume Kuma monitor for {service_name}: {e}")
                                results["errors"].append(f"Kuma resume failed for {service_name}: {str(e)}")
                        else:
                            results["errors"].append(f"Stack not found: {service_name}")
                    except Exception as e:
                        logger.error(f"Failed to start service {service_name}: {e}")
                        results["errors"].append(f"{service_name}: {str(e)}")
                success = len(results["started_services"]) > 0
                message = f"Started {len(results['started_services'])} service(s)"
                if results["errors"]:
                    message += f" with {len(results['errors'])} error(s)"
                return OperationResult(
                    success=success,
                    message=message,
                    details=results
                )
            except HTTPException:
                raise
            except Exception as e:
                logger.error(f"Failed to start service group '{name}': {e}")
                raise HTTPException(status_code=500, detail=str(e))
        # ===== Monitoring Endpoints =====
        @router.get(
            "/monitors",
            summary="List all monitors",
            response_model=Dict[str, Any]
        )
        async def list_monitors():
            """
            List all Uptime Kuma monitors
            Returns:
                List of monitors with their configurations
            """
            try:
                kuma = get_kuma_client()
                monitors = await kuma.get_monitors()
                return {
                    "success": True,
                    "monitors": monitors,
                    "total": len(monitors)
                }
            except Exception as e:
                logger.error(f"Failed to list monitors: {e}")
                raise HTTPException(status_code=500, detail=f"Failed to list monitors: {str(e)}")
        @router.post(
            "/monitors",
            summary="Create a new monitor",
            description="Create a new Uptime Kuma monitor. Requires admin authentication.",
            response_model=Dict[str, Any]
        )
        async def create_monitor(
            monitor_config: Dict[str, Any],
            user: Dict = Depends(get_admin_user)
        ):
            """
            Create a new Uptime Kuma monitor
            Args:
                monitor_config: Monitor configuration (name, type, hostname, port, etc.)
            Returns:
                Created monitor details including ID
            """
            try:
                kuma = get_kuma_client()
                created_monitor = await kuma.add_monitor(monitor_config)
                return {
                    "success": True,
                    "message": f"Monitor '{monitor_config.get('name')}' created successfully",
                    "monitor": created_monitor
                }
            except Exception as e:
                logger.error(f"Failed to create monitor: {e}")
                raise HTTPException(status_code=500, detail=f"Failed to create monitor: {str(e)}")
        @router.get(
            "/monitors/{monitor_id}",
            summary="Get monitor details",
            response_model=Dict[str, Any]
        )
        async def get_monitor(monitor_id: int):
            """
            Get details of a specific monitor
            Args:
                monitor_id: Monitor identifier
            Returns:
                Monitor configuration and status
            """
            try:
                kuma = get_kuma_client()
                monitor = await kuma.get_monitor(monitor_id)
                return {
                    "success": True,
                    "monitor": monitor
                }
            except Exception as e:
                logger.error(f"Failed to get monitor {monitor_id}: {e}")
                raise HTTPException(status_code=404, detail=f"Monitor {monitor_id} not found: {str(e)}")
        @router.put(
            "/monitors/{monitor_id}",
            summary="Update a monitor",
            description="Update an existing Uptime Kuma monitor. Requires admin authentication.",
            response_model=Dict[str, Any]
        )
        async def update_monitor(
            monitor_id: int,
            updates: Dict[str, Any],
            user: Dict = Depends(get_admin_user)
        ):
            """
            Update an existing monitor
            Args:
                monitor_id: Monitor identifier
                updates: Fields to update
            Returns:
                Updated monitor details
            """
            try:
                kuma = get_kuma_client()
                # Get existing monitor
                existing = await kuma.get_monitor(monitor_id)
                # Merge updates
                monitor_config = existing.copy()
                monitor_config.update(updates)
                # Update monitor
                updated_monitor = await kuma.update_monitor(monitor_id, monitor_config)
                return {
                    "success": True,
                    "message": f"Monitor {monitor_id} updated successfully",
                    "monitor": updated_monitor
                }
            except Exception as e:
                logger.error(f"Failed to update monitor {monitor_id}: {e}")
                raise HTTPException(status_code=500, detail=f"Failed to update monitor: {str(e)}")
        @router.delete(
            "/monitors/{monitor_id}",
            summary="Delete a monitor",
            description="Delete an Uptime Kuma monitor. Requires admin authentication.",
            response_model=Dict[str, Any]
        )
        async def delete_monitor(
            monitor_id: int,
            user: Dict = Depends(get_admin_user)
        ):
            """
            Delete a monitor
            Args:
                monitor_id: Monitor identifier
            Returns:
                Success confirmation
            """
            try:
                kuma = get_kuma_client()
                await kuma.delete_monitor(monitor_id)
                return {
                    "success": True,
                    "message": f"Monitor {monitor_id} deleted successfully"
                }
            except Exception as e:
                logger.error(f"Failed to delete monitor {monitor_id}: {e}")
                raise HTTPException(status_code=500, detail=f"Failed to delete monitor: {str(e)}")
        return router
@@ -0,0 +1,116 @@
 """
 Static Files Controller
 Serves static files for widgets and other frontend assets.
 """
 from fastapi import APIRouter
 from fastapi.responses import FileResponse, HTMLResponse
 from pathlib import Path
 import os
 from src.controllers.base import BaseController
 from src.logging_config import get_logger
 logger = get_logger(__name__)
 class StaticController(BaseController):
    """
    Controller for serving static files
    Provides endpoints for:
    - Organizr widgets
    - Other static assets
    """
    def __init__(self):
        super().__init__(prefix="/static", tags=["Static"])
        self.static_dir = Path(__file__).parent.parent.parent / "static"
    def create_router(self) -> APIRouter:
        """Create and configure the router"""
        router = APIRouter(prefix=self.prefix, tags=self.tags)
        @router.get(
            "/widgets/{filename}",
            response_class=HTMLResponse,
            summary="Get widget file"
        )
        async def get_widget(filename: str):
            """
            Serve widget HTML files
            Args:
                filename: Widget filename (e.g., service-control.html)
            Returns:
                HTML file content
            """
            widget_path = self.static_dir / "widgets" / filename
            if not widget_path.exists():
                return HTMLResponse(
                    content=f"<h1>404 - Widget not found</h1><p>{filename}</p>",
                    status_code=404
                )
            if not widget_path.is_file():
                return HTMLResponse(
                    content=f"<h1>400 - Not a file</h1>",
                    status_code=400
                )
            # Security: Ensure the path is within the static directory
            try:
                widget_path.resolve().relative_to(self.static_dir.resolve())
            except ValueError:
                return HTMLResponse(
                    content=f"<h1>403 - Forbidden</h1>",
                    status_code=403
                )
            logger.info(f"Serving widget: {filename}")
            return FileResponse(
                widget_path,
                media_type="text/html",
                headers={
                    "Cache-Control": "no-cache, no-store, must-revalidate",
                    "Pragma": "no-cache",
                    "Expires": "0"
                }
            )
        @router.get(
            "/widgets",
            summary="List available widgets"
        )
        async def list_widgets():
            """
            List all available widget files
            Returns:
                List of widget filenames
            """
            widgets_dir = self.static_dir / "widgets"
            if not widgets_dir.exists():
                return {"widgets": [], "message": "Widgets directory not found"}
            widgets = []
            for file in widgets_dir.glob("*.html"):
                widgets.append({
                    "name": file.name,
                    "url": f"/static/widgets/{file.name}",
                    "size": file.stat().st_size
                })
            return {
                "widgets": widgets,
                "count": len(widgets)
            }
        return router
 # Create controller instance
 static_controller = StaticController()
@@ -0,0 +1,98 @@
 """
 Tools Controller
 Provides utility tool endpoints including:
 - Web scraping and content extraction
 """
 from fastapi import APIRouter, HTTPException, status
 from src.controllers.base import BaseController
 from src.logging_config import get_logger
 from src.web_scraper.schemas import WebScraperRequest, WebScraperResponse
 from src.web_scraper.service import WebScraperService
 from src.web_scraper.exceptions import FetchError, ScrapingError
 logger = get_logger(__name__)
 class ToolsController(BaseController):
    """
    Controller for utility tools
    Provides endpoints for:
    - Web scraping and content extraction
    """
    def __init__(self):
        super().__init__(prefix="/web-scraper", tags=["Tools"])
        # Initialize service (could be dependency injected for testing)
        self.scraper_service = WebScraperService()
    def create_router(self) -> APIRouter:
        """Create and configure the router"""
        router = APIRouter(prefix=self.prefix, tags=self.tags)
        @router.post(
            "/scrape",
            response_model=WebScraperResponse,
            status_code=status.HTTP_200_OK,
            summary="Scrape website content",
            description="""
            Scrape and extract main content from a website.
            Uses trafilatura for intelligent content extraction (articles, blog posts, documentation),
            with BeautifulSoup as fallback. Perfect for feeding webpage content to LLMs.
            **Features:**
            - Intelligent main content extraction
            - Removes navigation, ads, footers
            - Optional link extraction
            - Configurable content length limits
            **Rate Limiting:** None (internal network use only)
            """
        )
        async def scrape_website(request: WebScraperRequest) -> WebScraperResponse:
            """
            Scrape a website and extract its main content
            Args:
                request: Scraping request with URL and options
            Returns:
                Extracted content with metadata
            Raises:
                HTTPException: 400 for fetch errors, 500 for processing errors
            """
            try:
                logger.info(f"Received scrape request for: {request.url}")
                result = await self.scraper_service.scrape_url(request)
                return result
            except FetchError as e:
                logger.warning(f"Fetch failed: {str(e)}")
                raise HTTPException(
                    status_code=status.HTTP_400_BAD_REQUEST,
                    detail=f"Failed to fetch URL: {str(e)}"
                )
            except ScrapingError as e:
                logger.error(f"Scraping failed: {str(e)}")
                raise HTTPException(
                    status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
                    detail=f"Failed to extract content: {str(e)}"
                )
            except Exception as e:
                logger.error(f"Unexpected error: {str(e)}", exc_info=True)
                raise HTTPException(
                    status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
                    detail="An unexpected error occurred"
                )
        return router
 # Create controller instance
 tools_controller = ToolsController()
@@ -8,12 +8,13 @@ from contextlib import asynccontextmanager
 from src.config import get_settings
 from src.logging_config import setup_logging, get_logger
 from src.web_scraper import router as web_scraper_router
 from src.api.v1.chat import router as chat_router
 from src.api.v1.models import router as models_router
 from src.api.v1.conversations import router as conversations_router
 from src.models.ollama_client import get_ollama_client, close_ollama_client
 from src.controllers.infrastructure_controller import infrastructure_controller
 from src.controllers.ai_controller import ai_controller
 from src.controllers.tools_controller import tools_controller
 from src.controllers.health_controller import health_controller
 from src.controllers.static_controller import static_controller
 from src.security import initialize_oidc
 # Initialize settings
 settings = get_settings()
@@ -47,6 +48,9 @@ async def lifespan(app: FastAPI):
    else:
        logger.warning("✗ Ollama connection failed - AI features may not work")
    # Initialize security (OIDC authentication)
    initialize_oidc(settings)
    yield
    # Shutdown
@@ -88,7 +92,7 @@ app = FastAPI(
    - `GET /infrastructure/ports` - List allocated ports
    - `GET /infrastructure/domains` - List configured domains
-    **Write Endpoints:**
+    **Write Endpoints (Admin Only):**
    - `POST /infrastructure/services` - Deploy new service from compose YAML
    - `PUT /infrastructure/services/{name}` - Update existing service
    - `DELETE /infrastructure/services/{name}` - Remove service and stack
@@ -100,6 +104,13 @@ app = FastAPI(
    Intelligent web scraping with main content extraction.
    Perfect for extracting articles, documentation, and blog posts for LLM consumption.
    ## Authentication
    When OIDC authentication is enabled (oidc_enabled=true in config):
    - Infrastructure write endpoints require authentication
    - Use OAuth2/OIDC bearer token from Authentik
    - Admin group membership required for infrastructure operations
    ## Integration
    This API is designed to integrate with:
@@ -118,7 +129,11 @@ app = FastAPI(
    redoc_url="/redoc",
    openapi_url="/openapi.json",
    lifespan=lifespan,
-    debug=settings.debug
+    debug=settings.debug,
    swagger_ui_init_oauth={
        "clientId": settings.oidc_audience,
        "usePkceWithAuthorizationCodeGrant": True,
    } if settings.oidc_enabled else None
 )
 # Add CORS middleware
@@ -131,69 +146,12 @@ app.add_middleware(
 )
-# Root endpoint
+# Include controller routers
-@app.get(
+app.include_router(health_controller.router)                   # / and /health
-    "/",
+app.include_router(ai_controller.router)                       # /v1/chat, /v1/models, /v1/conversations
-    tags=["Health"],
+app.include_router(tools_controller.router)                    # /web-scraper/scrape
-    summary="Service information",
+app.include_router(infrastructure_controller.router)           # /infrastructure/*
-    response_class=JSONResponse
+app.include_router(static_controller.router)                   # /static/*
 )
 async def root():
    """
    Get service information and health status
    Returns basic information about the API service and available endpoints.
    """
    logger.debug("Root endpoint accessed")
    return {
        "service": settings.app_name,
        "version": settings.app_version,
        "status": "healthy",
        "documentation": {
            "swagger_ui": "/docs",
            "redoc": "/redoc",
            "openapi_spec": "/openapi.json"
        },
        "endpoints": {
            "chat_completions": "/v1/chat/completions",
            "models": "/v1/models",
            "conversations": "/v1/conversations",
            "web_scraper": "/web-scraper/scrape",
            "infrastructure": "/infrastructure",
            "health": "/health"
        }
    }
 # Health check endpoint
@app.get(
    "/health",
    tags=["Health"],
    summary="Health check",
    response_class=JSONResponse
 )
 async def health_check():
    """
    Simple health check endpoint for container orchestration
    Returns a 200 OK status when the service is running properly.
    Used by Docker, Kubernetes, and load balancers.
    """
    ollama_client = get_ollama_client()
    ollama_healthy = await ollama_client.health_check()
    return {
        "status": "healthy",
        "ollama_connected": ollama_healthy
    }
 # Include routers
 app.include_router(chat_router)                              # /v1/chat/completions
 app.include_router(models_router)                            # /v1/models
 app.include_router(conversations_router)                     # /v1/conversations
 app.include_router(web_scraper_router)                       # /web-scraper/scrape
 app.include_router(infrastructure_controller.create_router())  # /infrastructure/*
 # Global exception handler
@@ -0,0 +1,32 @@
 """
 Security initialization module
 Handles OIDC configuration and authentication setup
 """
 from src.config import Settings
 from src.auth.oidc import oidc_config
 from src.logging_config import get_logger
 logger = get_logger(__name__)
 def initialize_oidc(settings: Settings) -> None:
    """
    Initialize OIDC authentication configuration
    Configures the global oidc_config instance with settings from environment.
    If OIDC is enabled, logs the issuer URL for verification.
    Args:
        settings: Application settings containing OIDC configuration
    """
    oidc_config.configure(
        enabled=settings.oidc_enabled,
        issuer=settings.oidc_issuer,
        audience=settings.oidc_audience
    )
    if settings.oidc_enabled:
        logger.info(f"✓ OIDC authentication enabled (issuer: {settings.oidc_issuer})")
    else:
        logger.info("○ OIDC authentication disabled - API is publicly accessible")
@@ -0,0 +1,135 @@
 """
 Service Groups and Safety Configuration
 Defines service groups, dependencies, and always-on infrastructure services.
 """
 from typing import List, Dict, Set
 # Always-on infrastructure services (CANNOT be stopped via API)
 ALWAYS_ON_SERVICES: Set[str] = {
    "portainer",
    "nginx-proxy-manager",
    "core-api",
    "uptime-kuma",
    "organizr",
    "headscale",
    "watchtower",
    "netdata",
    "maintenance",
 }
 # Service groups - services that should be started/stopped together
 SERVICE_GROUPS: Dict[str, List[str]] = {
    "jellyfin": [
        "jellyfin",
    ],
    "nextcloud": [
        "nextcloud",
        "nextcloud-db",
        "nextcloud-redis",
    ],
    "gitea": [
        "gitea",
        "gitea-db",
    ],
    "ai-stack": [
        "open-webui",
        "ollama",
        "qdrant",
    ],
    "samba": [
        "samba",
    ],
 }
 # Reverse mapping: service name -> group name
 SERVICE_TO_GROUP: Dict[str, str] = {}
 for group, services in SERVICE_GROUPS.items():
    for service in services:
        SERVICE_TO_GROUP[service] = group
 def is_always_on(service_name: str) -> bool:
    """
    Check if a service is marked as always-on (infrastructure)
    Args:
        service_name: Name of the service
    Returns:
        True if service cannot be stopped, False otherwise
    """
    return service_name.lower() in ALWAYS_ON_SERVICES
 def get_service_group(service_name: str) -> List[str]:
    """
    Get all services in the same group as the given service
    Args:
        service_name: Name of the service
    Returns:
        List of service names in the group (including the service itself)
        Returns [service_name] if not part of a group
    """
    group = SERVICE_TO_GROUP.get(service_name.lower())
    if group:
        return SERVICE_GROUPS[group].copy()
    return [service_name]
 def get_group_name(service_name: str) -> str:
    """
    Get the group name for a service
    Args:
        service_name: Name of the service
    Returns:
        Group name or the service name if not in a group
    """
    return SERVICE_TO_GROUP.get(service_name.lower(), service_name)
 def list_service_groups() -> Dict[str, List[str]]:
    """
    Get all defined service groups
    Returns:
        Dictionary of group names to service lists
    """
    return SERVICE_GROUPS.copy()
 def list_stoppable_services() -> List[str]:
    """
    Get list of all services that can be stopped
    Returns:
        List of service names that are not always-on
    """
    stoppable = []
    for services in SERVICE_GROUPS.values():
        stoppable.extend(services)
    # Remove any always-on services (shouldn't be in groups, but safety check)
    return [s for s in stoppable if not is_always_on(s)]
 def validate_stop_request(service_names: List[str]) -> tuple[bool, str]:
    """
    Validate that a list of services can be stopped
    Args:
        service_names: List of service names to check
    Returns:
        Tuple of (is_valid, error_message)
        error_message is empty string if valid
    """
    for service in service_names:
        if is_always_on(service):
            return False, f"Cannot stop always-on service: {service}"
    return True, ""
@@ -0,0 +1,343 @@
 <!DOCTYPE html>
 <html>
 <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Service Control</title>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }
        body {
            font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
            background: transparent;
            color: #e0e0e0;
            padding: 10px;
        }
        .container {
            max-width: 1200px;
            margin: 0 auto;
        }
        h2 {
            color: #fff;
            margin-bottom: 15px;
            font-size: 20px;
            font-weight: 500;
        }
        .service-grid {
            display: grid;
            grid-template-columns: repeat(auto-fill, minmax(300px, 1fr));
            gap: 15px;
        }
        .service-card {
            background: rgba(40, 40, 40, 0.95);
            border: 1px solid rgba(255, 255, 255, 0.1);
            border-radius: 8px;
            padding: 15px;
            transition: all 0.3s ease;
        }
        .service-card:hover {
            border-color: rgba(66, 153, 225, 0.5);
            box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
        }
        .service-header {
            display: flex;
            justify-content: space-between;
            align-items: center;
            margin-bottom: 12px;
        }
        .service-name {
            font-size: 16px;
            font-weight: 600;
            color: #fff;
            text-transform: capitalize;
        }
        .status-badge {
            padding: 4px 12px;
            border-radius: 12px;
            font-size: 12px;
            font-weight: 600;
            text-transform: uppercase;
        }
        .status-running {
            background: rgba(72, 187, 120, 0.2);
            color: #48bb78;
            border: 1px solid rgba(72, 187, 120, 0.4);
        }
        .status-stopped {
            background: rgba(245, 101, 101, 0.2);
            color: #f56565;
            border: 1px solid rgba(245, 101, 101, 0.4);
        }
        .status-loading {
            background: rgba(237, 137, 54, 0.2);
            color: #ed8936;
            border: 1px solid rgba(237, 137, 54, 0.4);
        }
        .service-info {
            font-size: 13px;
            color: #a0a0a0;
            margin-bottom: 12px;
        }
        .service-actions {
            display: flex;
            gap: 8px;
        }
        .btn {
            flex: 1;
            padding: 8px 12px;
            border: none;
            border-radius: 6px;
            font-size: 13px;
            font-weight: 600;
            cursor: pointer;
            transition: all 0.2s ease;
            text-transform: uppercase;
            letter-spacing: 0.5px;
        }
        .btn:disabled {
            opacity: 0.5;
            cursor: not-allowed;
        }
        .btn-start {
            background: linear-gradient(135deg, #48bb78 0%, #38a169 100%);
            color: white;
        }
        .btn-start:hover:not(:disabled) {
            background: linear-gradient(135deg, #38a169 0%, #2f855a 100%);
            transform: translateY(-1px);
        }
        .btn-stop {
            background: linear-gradient(135deg, #f56565 0%, #e53e3e 100%);
            color: white;
        }
        .btn-stop:hover:not(:disabled) {
            background: linear-gradient(135deg, #e53e3e 0%, #c53030 100%);
            transform: translateY(-1px);
        }
        .btn-restart {
            background: linear-gradient(135deg, #4299e1 0%, #3182ce 100%);
            color: white;
        }
        .btn-restart:hover:not(:disabled) {
            background: linear-gradient(135deg, #3182ce 0%, #2c5282 100%);
            transform: translateY(-1px);
        }
        .loading {
            text-align: center;
            padding: 40px;
            color: #a0a0a0;
        }
        .error {
            background: rgba(245, 101, 101, 0.1);
            border: 1px solid rgba(245, 101, 101, 0.4);
            color: #f56565;
            padding: 12px;
            border-radius: 6px;
            margin-bottom: 15px;
        }
        .always-on-badge {
            display: inline-block;
            padding: 2px 8px;
            background: rgba(66, 153, 225, 0.2);
            color: #4299e1;
            border: 1px solid rgba(66, 153, 225, 0.4);
            border-radius: 10px;
            font-size: 11px;
            margin-left: 8px;
        }
        @keyframes spin {
            to { transform: rotate(360deg); }
        }
        .spinner {
            display: inline-block;
            width: 14px;
            height: 14px;
            border: 2px solid rgba(255, 255, 255, 0.3);
            border-top-color: #fff;
            border-radius: 50%;
            animation: spin 0.6s linear infinite;
            margin-right: 6px;
        }
    </style>
 </head>
 <body>
    <div class="container">
        <h2>🎛️ On-Demand Services</h2>
        <div id="error-container"></div>
        <div id="service-container" class="loading">Loading services...</div>
    </div>
    <script>
        // Auto-detect API base from current domain (works with NPM proxy)
        const API_BASE = window.location.origin;
        let services = [];
        let alwaysOnServices = [];
        async function fetchServices() {
            try {
                const response = await fetch(`${API_BASE}/infrastructure/services`);
                if (!response.ok) throw new Error('Failed to fetch services');
                services = await response.json();
                const groupsResponse = await fetch(`${API_BASE}/infrastructure/service-groups`);
                if (groupsResponse.ok) {
                    const groupsData = await groupsResponse.json();
                    alwaysOnServices = groupsData.always_on || [];
                }
                renderServices();
                document.getElementById('error-container').innerHTML = '';
            } catch (error) {
                console.error('Error fetching services:', error);
                document.getElementById('error-container').innerHTML =
                    `<div class="error">❌ Failed to connect to API: ${error.message}</div>`;
            }
        }
        function isAlwaysOn(serviceName) {
            return alwaysOnServices.includes(serviceName.toLowerCase());
        }
        function getServiceStatus(service) {
            if (service.containers_running > 0) {
                return {
                    class: 'status-running',
                    text: `Running (${service.containers_running}/${service.containers_total})`
                };
            } else if (service.containers_total > 0) {
                return {
                    class: 'status-stopped',
                    text: 'Stopped'
                };
            } else {
                return {
                    class: 'status-stopped',
                    text: 'No containers'
                };
            }
        }
        function renderServices() {
            const container = document.getElementById('service-container');
            // Filter to only show stoppable services
            const stoppableServices = services.filter(s => !isAlwaysOn(s.name));
            if (stoppableServices.length === 0) {
                container.innerHTML = '<div class="loading">No stoppable services found</div>';
                return;
            }
            container.className = 'service-grid';
            container.innerHTML = stoppableServices.map(service => {
                const status = getServiceStatus(service);
                const isRunning = service.containers_running > 0;
                const alwaysOn = isAlwaysOn(service.name);
                return `
                    <div class="service-card" data-service="${service.name}">
                        <div class="service-header">
                            <span class="service-name">
                                ${service.name}
                                ${alwaysOn ? '<span class="always-on-badge">ALWAYS ON</span>' : ''}
                            </span>
                            <span class="status-badge ${status.class}">${status.text}</span>
                        </div>
                        <div class="service-info">
                            Stack ID: ${service.stack_id || 'N/A'}
                        </div>
                        <div class="service-actions">
                            <button class="btn btn-start"
                                    onclick="controlService('${service.name}', 'start')"
                                    ${isRunning || alwaysOn ? 'disabled' : ''}>
                                Start
                            </button>
                            <button class="btn btn-stop"
                                    onclick="controlService('${service.name}', 'stop')"
                                    ${!isRunning || alwaysOn ? 'disabled' : ''}>
                                Stop
                            </button>
                        </div>
                    </div>
                `;
            }).join('');
        }
        async function controlService(serviceName, action) {
            const card = document.querySelector(`[data-service="${serviceName}"]`);
            const buttons = card.querySelectorAll('button');
            // Disable all buttons and show loading
            buttons.forEach(btn => {
                btn.disabled = true;
                if (btn.textContent.toLowerCase().includes(action)) {
                    btn.innerHTML = `<span class="spinner"></span>${action.toUpperCase()}...`;
                }
            });
            try {
                const response = await fetch(`${API_BASE}/infrastructure/services/${serviceName}/${action}`, {
                    method: 'POST'
                });
                const result = await response.json();
                if (!response.ok || !result.success) {
                    throw new Error(result.message || result.detail || 'Operation failed');
                }
                console.log(`${action} ${serviceName}:`, result);
                // Wait a bit for containers to start/stop
                await new Promise(resolve => setTimeout(resolve, 2000));
                // Refresh service list
                await fetchServices();
            } catch (error) {
                console.error(`Error ${action}ing ${serviceName}:`, error);
                alert(`Failed to ${action} ${serviceName}: ${error.message}`);
                // Re-enable buttons on error
                await fetchServices();
            }
        }
        // Auto-refresh every 10 seconds
        setInterval(fetchServices, 10000);
        // Initial load
        fetchServices();
    </script>
 </body>
 </html>
@@ -0,0 +1,54 @@
 #!/usr/bin/env python3
 """
 Test what headers are being sent to Organizr
 """
 import asyncio
 import sys
 from pathlib import Path
 sys.path.insert(0, str(Path(__file__).parent))
 from src.clients.npm_client import get_npm_client
 async def main():
    npm = get_npm_client()
    # Find home.schweitz.net
    hosts = await npm.get_proxy_hosts()
    for host in hosts:
        if 'home.schweitz.net' in host.get('domain_names', []):
            print(f"Found: {', '.join(host.get('domain_names', []))}")
            print(f"Forward to: {host.get('forward_scheme')}://{host.get('forward_host')}:{host.get('forward_port')}")
            print()
            config = host.get('advanced_config', '')
            print("Checking for Authentik headers in nginx config:")
            print("=" * 60)
            headers_to_check = [
                'X-authentik-username',
                'X-authentik-email',
                'X-authentik-groups',
                'X-authentik-name',
                'X-authentik-uid'
            ]
            for header in headers_to_check:
                if f'proxy_set_header {header}' in config:
                    print(f"✓ {header} is configured")
                else:
                    print(f"✗ {header} is NOT configured")
            print()
            print("Full advanced config:")
            print("=" * 60)
            print(config)
            break
 if __name__ == "__main__":
    asyncio.run(main())
@@ -0,0 +1,115 @@
 #!/usr/bin/env python3
 """
 Update all NPM proxy hosts to use port 9001 for Authentik forward auth
 """
 import asyncio
 import httpx
 import os
 NPM_URL = os.getenv("NPM_URL", "http://192.168.86.149:81")
 NPM_EMAIL = os.getenv("NPM_EMAIL", "admin@example.com")
 NPM_PASSWORD = os.getenv("NPM_PASSWORD", "changeme")
 async def get_npm_token():
    """Get NPM authentication token"""
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{NPM_URL}/api/tokens",
            json={"identity": NPM_EMAIL, "secret": NPM_PASSWORD}
        )
        response.raise_for_status()
        return response.json()["token"]
 async def get_proxy_hosts(token):
    """Get all proxy hosts"""
    headers = {"Authorization": f"Bearer {token}"}
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"{NPM_URL}/api/nginx/proxy-hosts",
            headers=headers
        )
        response.raise_for_status()
        return response.json()
 async def update_proxy_host(token, host_id, config):
    """Update a proxy host"""
    headers = {"Authorization": f"Bearer {token}"}
    async with httpx.AsyncClient() as client:
        response = await client.put(
            f"{NPM_URL}/api/nginx/proxy-hosts/{host_id}",
            headers=headers,
            json=config
        )
        response.raise_for_status()
        return response.json()
 async def main():
    print("Updating NPM proxy hosts to use port 9001...\n")
    # Get token
    token = await get_npm_token()
    # Get all proxy hosts
    hosts = await get_proxy_hosts(token)
    updated = []
    skipped = []
    for host in hosts:
        host_id = host.get("id")
        domain_names = host.get("domain_names", [])
        domain_str = ", ".join(domain_names)
        advanced_config = host.get("advanced_config", "")
        # Skip if no authentik config
        if "authentik" not in advanced_config.lower():
            continue
        # Skip if already port 9001
        if ":9001" in advanced_config:
            print(f"⊘ {domain_str} - Already using port 9001")
            skipped.append(domain_str)
            continue
        # Update 9000 to 9001
        if ":9000" in advanced_config:
            print(f"⟳ {domain_str} - Updating to port 9001...", end=" ")
            new_config = advanced_config.replace(":9000", ":9001")
            # Clean config
            readonly_fields = [
                "id", "created_on", "modified_on", "owner", "owner_user_id",
                "certificate", "use_default_location", "ipv6", "meta",
                "nginx_online", "nginx_err", "access_list", "certificate_id"
            ]
            clean_host = {k: v for k, v in host.items() if k not in readonly_fields}
            clean_host["advanced_config"] = new_config
            if "locations" not in clean_host or clean_host["locations"] is None:
                clean_host["locations"] = []
            try:
                await update_proxy_host(token, host_id, clean_host)
                print("✓")
                updated.append(domain_str)
            except Exception as e:
                print(f"✗ Error: {e}")
    print(f"\n{'='*60}")
    print(f"Updated: {len(updated)} hosts")
    print(f"Skipped: {len(skipped)} hosts")
    if updated:
        print("\nUpdated hosts:")
        for d in updated:
            print(f"  • {d}")
 if __name__ == "__main__":
    asyncio.run(main())
@@ -0,0 +1,194 @@
 version: '3.8'
 # Authentik - Identity Provider for SSO (Using Shared Infrastructure)
 # Phase 1: Foundation - Google OAuth Integration
 # Ports: 9000 (HTTP), 9443 (HTTPS)
 # GPU: No
 # Dependencies: postgres-shared, redis-shared
 services:
  authentik-server:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik-server
    restart: unless-stopped
    command: server
    ports:
      - "9000:9000"
      # Port 9443 removed - use NPM for HTTPS termination
    environment:
      # Database configuration (shared PostgreSQL)
      AUTHENTIK_POSTGRESQL__HOST: postgres-shared
      AUTHENTIK_POSTGRESQL__PORT: 5432
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__USER: authentik_user
      AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_DB_PASSWORD:?database password required}
      # Cache configuration (shared Redis, database 1)
      AUTHENTIK_REDIS__HOST: redis-shared
      AUTHENTIK_REDIS__PORT: 6379
      AUTHENTIK_REDIS__DB: 1
      # Authentik secret key
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:?secret key required}
      # Error reporting (disabled)
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
      # Performance tuning for home use
      WORKERS: 2
      # Email configuration (optional - configure later if needed)
      # AUTHENTIK_EMAIL__HOST: smtp.gmail.com
      # AUTHENTIK_EMAIL__PORT: 587
      # AUTHENTIK_EMAIL__USERNAME: your-email@gmail.com
      # AUTHENTIK_EMAIL__PASSWORD: your-app-password
      # AUTHENTIK_EMAIL__USE_TLS: "true"
      # AUTHENTIK_EMAIL__FROM: authentik@schweitz.net
      # Timezone
      TZ: Europe/Amsterdam
    volumes:
      - /home/jpmschweitzer/docker-data/authentik/media:/media
      - /home/jpmschweitzer/docker-data/authentik/custom-templates:/templates
    networks:
      - docker-dataplane
    depends_on:
      - postgres-shared
      - redis-shared
    deploy:
      resources:
        limits:
          memory: 256M
  authentik-worker:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik-worker
    restart: unless-stopped
    command: worker
    environment:
      # Database configuration (shared PostgreSQL)
      AUTHENTIK_POSTGRESQL__HOST: postgres-shared
      AUTHENTIK_POSTGRESQL__PORT: 5432
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__USER: authentik_user
      AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_DB_PASSWORD}
      # Cache configuration (shared Redis, database 1)
      AUTHENTIK_REDIS__HOST: redis-shared
      AUTHENTIK_REDIS__PORT: 6379
      AUTHENTIK_REDIS__DB: 1
      # Authentik secret key
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      # Error reporting (disabled)
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
      # Timezone
      TZ: Europe/Amsterdam
    user: root
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/jpmschweitzer/docker-data/authentik/media:/media
      - /home/jpmschweitzer/docker-data/authentik/certs:/certs
      - /home/jpmschweitzer/docker-data/authentik/custom-templates:/templates
    networks:
      - docker-dataplane
    depends_on:
      - postgres-shared
      - redis-shared
    deploy:
      resources:
        limits:
          memory: 192M
  authentik-proxy-outpost:
    image: ghcr.io/goauthentik/proxy:latest
    container_name: authentik-proxy-outpost
    restart: unless-stopped
    network_mode: host
    environment:
      # Authentik connection
      AUTHENTIK_HOST: http://192.168.86.149:9000
      AUTHENTIK_INSECURE: "false"
      AUTHENTIK_TOKEN: ${AUTHENTIK_OUTPOST_TOKEN:?outpost token required}
      # Logging
      AUTHENTIK_LOG_LEVEL: info
      # Port configuration
      AUTHENTIK_LISTEN__HTTP: 0.0.0.0:9001
      AUTHENTIK_LISTEN__METRICS: 0.0.0.0:9300
    depends_on:
      - authentik-server
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:9001/outpost.goauthentik.io/ping"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 # Prerequisites:
 #
 # 1. Deploy shared-infrastructure stack first!
 #    docker-compose -f shared-infrastructure.yml up -d
 #
 # 2. Verify shared services are running:
 #    docker ps | grep -E 'postgres-shared|redis-shared'
 #
 # 3. Create Authentik data directories (if not exists):
 #    mkdir -p ~/docker-data/authentik/{media,certs,custom-templates}
 #
 # 4. Create .env file with:
 #    AUTHENTIK_DB_PASSWORD=<from shared-infrastructure .env>
 #    AUTHENTIK_SECRET_KEY=<generate with: openssl rand -base64 60>
 #
 # 5. Deploy this stack:
 #    docker-compose -f authentik-shared.yml --env-file .env.authentik-shared up -d
 #
 # After Deployment:
 #
 # 1. Wait for containers to start (may take 30-60 seconds for DB migrations)
 #
 # 2. Check logs:
 #    docker logs authentik-server
 #    docker logs authentik-worker
 #
 # 3. Access initial setup: http://localhost:9000/if/flow/initial-setup/
 #    - Create admin account (akadmin recommended)
 #    - Set strong password
 #
 # 4. Configure NPM reverse proxy:
 #    - Domain: auth.schweitz.net
 #    - Forward to: authentik-server:9000
 #    - SSL: Let's Encrypt
 #    - Websockets: Enabled
 #
 # 5. Access admin interface: https://auth.schweitz.net/if/admin/
 #
 # Connection Details:
 #
 # Database:
 #   - Host: postgres-shared (from containers) / localhost (from host)
 #   - Port: 5432
 #   - Database: authentik
 #   - User: authentik_user
 #
 # Cache:
 #   - Host: redis-shared (from containers) / localhost (from host)
 #   - Port: 6379
 #   - Database: 1
 #
 # Resource Usage (optimized for home use):
 #   - Server: 256MB RAM limit (WORKERS=2 reduces Gunicorn processes)
 #   - Worker: 192MB RAM limit
 #   - Proxy Outpost: ~32MB RAM
 #   - Total Authentik: ~480MB max (vs ~700MB with dedicated PostgreSQL/Redis)
 #   - Savings: ~400MB by using shared infrastructure!
@@ -3,7 +3,7 @@ version: '3.8'
 # Core API - OpenAPI-compatible functions and AI orchestration for Open WebUI
 # Purpose: Provides OpenAI-compatible API (/v1/chat/completions) and tool functions (web scraping)
 # Port: 8083 (HTTP API)
-# Network: ai-dataplane (shared with Open WebUI, Ollama, Qdrant)
+# Network: docker-dataplane (shared infrastructure network)
 #
 # Setup: Create venv before first deployment:
 #   cd /home/jpmschweitzer/Projects/portainer-core/services/core-api
@@ -17,7 +17,7 @@ services:
    container_name: core-api
    restart: unless-stopped
-    # Hot-reload development mode
+    # Production mode with workers
    command: >
      sh -c "
      if [ ! -f /venv/bin/activate ]; then
@@ -29,8 +29,7 @@ services:
      /venv/bin/uvicorn src.main:app
      --host 0.0.0.0
      --port 8083
-      --reload
+      --workers 2
      --reload-dir /app/src
      "
    ports:
@@ -72,6 +71,11 @@ services:
      - WEB_SCRAPER_DEFAULT_MAX_LENGTH=10000
      - WEB_SCRAPER_MAX_LINKS_TO_EXTRACT=50
      # Uptime Kuma Configuration
      - KUMA_URL=http://uptime-kuma:3001
      - KUMA_USERNAME=${KUMA_USERNAME}
      - KUMA_PASSWORD=${KUMA_PASSWORD}
      # Python path
      - PYTHONPATH=/app
@@ -86,7 +90,15 @@ services:
      - /home/jpmschweitzer/docker-data/core-api/logs:/app/logs
    networks:
-      - ai-dataplane
+      - docker-dataplane
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 2G
        reservations:
          memory: 512M
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
@@ -99,5 +111,6 @@ services:
      start_period: 30s
 networks:
-  ai-dataplane:
+  docker-dataplane:
    external: true
    name: docker-dataplane
@@ -20,7 +20,7 @@ services:
      - POSTGRES_DB=gitea
      - TZ=Europe/Amsterdam
    networks:
-      - gitea-network
+      - docker-dataplane
  gitea:
    image: gitea/gitea:latest
@@ -46,11 +46,12 @@ services:
    depends_on:
      - gitea-db
    networks:
-      - gitea-network
+      - docker-dataplane
 networks:
-  gitea-network:
+  docker-dataplane:
-    driver: bridge
+    external: true
    name: docker-dataplane
 # ⚠️ SECURITY WARNING:
 # Change POSTGRES_PASSWORD and GITEA__database__PASSWD before deploying!
@@ -21,11 +21,12 @@ services:
    environment:
      - TZ=Europe/Amsterdam
    networks:
-      - headscale-network
+      - docker-dataplane
 networks:
-  headscale-network:
+  docker-dataplane:
-    driver: bridge
+    external: true
    name: docker-dataplane
 # Setup Instructions:
 # 1. Create directories:
@@ -1,43 +0,0 @@
 version: '3.8'
 # Heimdall - Application Dashboard
 # Phase 3: Monitoring & Management
 # Ports: 8888 (HTTP), 8889 (HTTPS)
 # GPU: No
 # Storage: SSD (configuration)
 services:
  heimdall:
    image: linuxserver/heimdall:latest
    container_name: heimdall
    restart: unless-stopped
    ports:
      - "8888:80"
      - "8889:443"
    volumes:
      - /home/jpmschweitzer/docker-data/heimdall:/config
    environment:
      - PUID=1000  # Your user ID (run: id -u)
      - PGID=1000  # Your group ID (run: id -g)
      - TZ=Europe/Amsterdam
 # After Deployment:
 # 1. Access http://localhost:8888
 # 2. Add application tiles for quick access:
 #    - Portainer: http://tower-of-joy:8080
 #    - NPM: http://tower-of-joy:8000
 #    - Jellyfin: http://tower-of-joy:8096
 #    - Nextcloud: http://tower-of-joy:8082
 #    - AMP: http://tower-of-joy:8081
 #    - Uptime Kuma: http://tower-of-joy:3001
 #    - Netdata: http://tower-of-joy:19999
 #    - Ollama: http://tower-of-joy:11434
 # 3. Customize colors and icons for each service
 # 4. Set as browser homepage for easy access
 #
 # Features:
 # - Unified dashboard for all services
 # - One-click access to any service
 # - Custom backgrounds and themes
 # - Search functionality
 # - Mobile-friendly
@@ -0,0 +1,64 @@
 #!/bin/bash
 set -e
 # PostgreSQL Initialization Script
 # Creates databases and users for homelab applications
 # Runs once during initial container startup
 echo "🔧 Initializing PostgreSQL databases and users..."
 # Source environment variables if available
 if [ -f /run/secrets/postgres_passwords ]; then
    source /run/secrets/postgres_passwords
 fi
 # Default passwords (override via environment variables)
 : ${AUTHENTIK_DB_PASSWORD:=CHANGEME_AUTHENTIK_PASSWORD}
 : ${GITEA_DB_PASSWORD:=CHANGEME_GITEA_PASSWORD}
 # Create Authentik database and user
 echo "📦 Creating Authentik database..."
 psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
    CREATE DATABASE authentik;
    CREATE USER authentik_user WITH PASSWORD '$AUTHENTIK_DB_PASSWORD';
    GRANT ALL PRIVILEGES ON DATABASE authentik TO authentik_user;
    -- Grant schema privileges (required for PostgreSQL 15+)
    \c authentik
    GRANT ALL ON SCHEMA public TO authentik_user;
    GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO authentik_user;
    GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO authentik_user;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO authentik_user;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO authentik_user;
 EOSQL
 # Create Gitea database and user (for future migration)
 echo "📦 Creating Gitea database..."
 psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
    CREATE DATABASE gitea;
    CREATE USER gitea_user WITH PASSWORD '$GITEA_DB_PASSWORD';
    GRANT ALL PRIVILEGES ON DATABASE gitea TO gitea_user;
    \c gitea
    GRANT ALL ON SCHEMA public TO gitea_user;
    GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO gitea_user;
    GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO gitea_user;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO gitea_user;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO gitea_user;
 EOSQL
 # Add future databases here as needed
 # echo "📦 Creating future_app database..."
 # psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
 #     CREATE DATABASE future_app;
 #     CREATE USER future_user WITH PASSWORD '$FUTURE_APP_PASSWORD';
 #     GRANT ALL PRIVILEGES ON DATABASE future_app TO future_user;
 # EOSQL
 echo "✅ PostgreSQL initialization complete!"
 echo ""
 echo "📊 Database Summary:"
 psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" -c '\l'
 echo ""
 echo "👥 User Summary:"
 psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" -c '\du'
@@ -42,6 +42,13 @@ services:
    # crond flags:
    #   -f: foreground (don't daemonize)
    #   -l 2: log level 2 (errors and info)
    networks:
      - docker-dataplane
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 # Usage:
 # 1. Create maintenance scripts in ~/docker-data/maintenance/scripts/
@@ -27,6 +27,13 @@ services:
      # Optional: Claim to Netdata Cloud for remote access
      # - NETDATA_CLAIM_TOKEN=your-claim-token
      # - NETDATA_CLAIM_URL=https://app.netdata.cloud
    networks:
      - docker-dataplane
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 # After Deployment:
 # 1. Access http://localhost:19999
@@ -0,0 +1,398 @@
 # Nextcloud Database Consolidation Plan
 **Goal**: Fresh Nextcloud installation using shared PostgreSQL + Redis infrastructure
 **Date**: 2025-11-16
 **Status**: APPROVED - Complete wipe and fresh start
 **Approach**: No migration, no backups - complete fresh installation
 ---
 ## Current State
 ### Existing Setup
 ```yaml
 nextcloud-db (MariaDB 10.11)
  ├─ Database: nextcloud
  ├─ User: nextcloud
  ├─ Data: /home/jpmschweitzer/docker-data/nextcloud/db
  └─ Network: docker-dataplane
 nextcloud-redis (Redis Alpine)
  ├─ Standalone instance
  ├─ Data: In-memory only (no persistence configured)
  └─ Network: docker-dataplane
 nextcloud (Nextcloud Stable)
  ├─ Config: /home/jpmschweitzer/docker-data/nextcloud/config
  ├─ Data: /mnt/media/nextcloud/data
  └─ Dependencies: nextcloud-db, nextcloud-redis
 ```
 ### Target Setup
 ```yaml
 postgres-shared (PostgreSQL 16)
  ├─ New database: nextcloud
  ├─ New user: nextcloud_user
  └─ Database allocation: DB 3
 redis-shared (Redis Alpine)
  ├─ Database allocation: DB 3 (Nextcloud)
  ├─ Existing DB 0: General cache
  ├─ Existing DB 1: Authentik
  └─ Existing DB 2: Gitea
 ```
 ---
 ## Migration Challenges
 ### Critical Issue: MariaDB → PostgreSQL
 ⚠️ **Nextcloud cannot simply switch database types!**
 Nextcloud's database schema is different between MariaDB and PostgreSQL:
 - Different data types (e.g., LONGTEXT vs TEXT)
 - Different auto-increment handling
 - Different JSON field types
 - Different index structures
 **Options:**
 ### Option A: Fresh Install + Data Migration (RECOMMENDED)
 ✅ **Pros:**
 - Clean database schema
 - Opportunity to optimize
 - Lower risk of corruption
 - Can test before switching
 ❌ **Cons:**
 - Must recreate users/settings
 - Requires careful data migration
 - More complex process
 ### Option B: Database Conversion
 ✅ **Pros:**
 - Preserves all settings
 - Preserves user data
 ❌ **Cons:**
 - Complex conversion process
 - High risk of data loss
 - Nextcloud doesn't officially support this
 - May leave corrupted data
 **RECOMMENDATION: Option A (Fresh Install)**
 ---
 ## Fresh Installation Plan
 ### Phase 1: Complete Cleanup - PURGE ALL DATA
 **Estimated Time:** 2 minutes
 ⚠️ **DESTRUCTIVE OPERATION - REQUIRES EXPLICIT APPROVAL** ⚠️
 The following will be PERMANENTLY DELETED:
 - All Nextcloud containers (nextcloud, nextcloud-db, nextcloud-redis)
 - All Nextcloud configuration (/home/jpmschweitzer/docker-data/nextcloud)
 - All Nextcloud user files (/mnt/media/nextcloud)
 - All Nextcloud database data
 **APPROVAL REQUIRED BEFORE EACH DELETION STEP**
 ```bash
 # Step 1: Stop and remove containers
 # APPROVAL: Stop containers? (y/n)
 docker stop nextcloud nextcloud-db nextcloud-redis 2>/dev/null || true
 docker rm nextcloud nextcloud-db nextcloud-redis 2>/dev/null || true
 # Step 2: Delete config directory
 # APPROVAL: Delete /home/jpmschweitzer/docker-data/nextcloud? (y/n)
 sudo rm -rf /home/jpmschweitzer/docker-data/nextcloud
 # Step 3: Delete user data directory
 # APPROVAL: Delete /mnt/media/nextcloud? (y/n)
 sudo rm -rf /mnt/media/nextcloud
 # Step 4: Verify complete removal
 ls /home/jpmschweitzer/docker-data/ | grep nextcloud  # Should be empty
 ls /mnt/media/ | grep nextcloud  # Should be empty
 ```
 ### Phase 2: Prepare Shared Infrastructure
 **Estimated Time:** 5 minutes
 ```bash
 # 1. Create Nextcloud database in postgres-shared
 docker exec -i postgres-shared psql -U postgres <<'EOF'
 -- Nextcloud database
 CREATE DATABASE nextcloud;
 CREATE USER nextcloud_user WITH PASSWORD 'GENERATE_NEW_PASSWORD_HERE';
 GRANT ALL PRIVILEGES ON DATABASE nextcloud TO nextcloud_user;
 \c nextcloud
 GRANT ALL ON SCHEMA public TO nextcloud_user;
 ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO nextcloud_user;
 ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO nextcloud_user;
 EOF
 # 2. Update redis-shared documentation (already supports DB 3)
 # No action needed - redis-shared already configured for multi-database
 ```
 ### Phase 3: Create Fresh Nextcloud Stack Configuration
 **Estimated Time:** 5 minutes
 Create new `nextcloud-shared.yml`:
 ```yaml
 services:
  nextcloud:
    image: nextcloud:stable
    container_name: nextcloud
    restart: unless-stopped
    ports:
      - "8082:80"
    volumes:
      # Fresh config directory
      - /home/jpmschweitzer/docker-data/nextcloud/config:/var/www/html/config
      # Fresh user data directory
      - /mnt/media/nextcloud/data:/var/www/html/data
    environment:
      # PostgreSQL configuration
      - POSTGRES_HOST=postgres-shared
      - POSTGRES_DB=nextcloud
      - POSTGRES_USER=nextcloud_user
      - POSTGRES_PASSWORD=${NEXTCLOUD_DB_PASSWORD}
      # Redis configuration (Database 3)
      - REDIS_HOST=redis-shared
      - REDIS_HOST_PORT=6379
      - REDIS_DB_INDEX=3
      # Timezone
      - TZ=Europe/Amsterdam
    depends_on:
      - postgres-shared
      - redis-shared
    networks:
      - docker-dataplane
    deploy:
      resources:
        limits:
          memory: 1G
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 ```
 ### Phase 5: Deploy Fresh Nextcloud
 **Estimated Time:** 5 minutes
 ```bash
 # 1. Create new config directory
 mkdir -p /home/jpmschweitzer/docker-data/nextcloud-shared/config
 # 2. Create .env file with database password
 cat > /mnt/media/Projects/portainer-core/stacks/.env.nextcloud-shared <<EOF
 NEXTCLOUD_DB_PASSWORD=<GENERATED_PASSWORD_FROM_PHASE2>
 EOF
 # 3. Deploy new stack
 cd /mnt/media/Projects/portainer-core/stacks
 docker compose -f nextcloud-shared.yml --env-file .env.nextcloud-shared up -d
 # 4. Wait for initialization
 docker logs -f nextcloud
 ```
 ### Phase 6: Initial Setup & Configuration
 **Estimated Time:** 10 minutes
 ```bash
 # 1. Access Nextcloud web interface
 # Navigate to: http://localhost:8082 or https://cloud.schweitz.net
 # 2. First-time setup wizard:
 #    - Admin username: admin
 #    - Admin password: <STRONG_PASSWORD>
 #    - Data folder: /var/www/html/data (default)
 #    - Database: PostgreSQL
 #    - Database user: nextcloud_user
 #    - Database password: <FROM_ENV_FILE>
 #    - Database name: nextcloud
 #    - Database host: postgres-shared
 # 3. Wait for installation (2-3 minutes)
 # 4. Configure trusted domains
 docker exec -u www-data nextcloud php occ config:system:set trusted_domains 1 --value=cloud.schweitz.net
 docker exec -u www-data nextcloud php occ config:system:set trusted_domains 2 --value=192.168.86.149
 # 5. Configure Redis caching
 docker exec -u www-data nextcloud php occ config:system:set redis host --value=redis-shared
 docker exec -u www-data nextcloud php occ config:system:set redis port --value=6379
 docker exec -u www-data nextcloud php occ config:system:set redis dbindex --value=3
 docker exec -u www-data nextcloud php occ config:system:set memcache.local --value='\\OC\\Memcache\\APCu'
 docker exec -u www-data nextcloud php occ config:system:set memcache.distributed --value='\\OC\\Memcache\\Redis'
 docker exec -u www-data nextcloud php occ config:system:set memcache.locking --value='\\OC\\Memcache\\Redis'
 # 6. Optimize database
 docker exec -u www-data nextcloud php occ db:add-missing-indices
 docker exec -u www-data nextcloud php occ db:convert-filecache-bigint
 # 7. Configure background jobs
 docker exec -u www-data nextcloud php occ background:cron
 ```
 ### Phase 7: Verify Fresh Installation
 **Estimated Time:** 5 minutes
 ```bash
 # 1. Verify admin user can login via web interface
 # Navigate to: https://cloud.schweitz.net
 # 2. Check PostgreSQL connection
 docker exec postgres-shared psql -U nextcloud_user -d nextcloud -c '\dt'
 # 3. Check Redis caching
 docker exec redis-shared redis-cli -n 3 DBSIZE
 # 4. Verify storage location
 docker exec -u www-data nextcloud php occ config:system:get datadirectory
 # 5. Test file upload/download
 # Upload a test file via web interface
 # Download it back
 # Delete it
 ```
 ### Phase 8: Final Cleanup & Documentation
 **Estimated Time:** 2 minutes
 ```bash
 # 1. Update postgres-shared.yml documentation
 # Add Nextcloud to "Applications Using This Database" list
 # 2. Update redis-shared.yml documentation
 # Add "DB 3: Nextcloud (file locking, distributed cache)"
 # 3. Rename stack file
 cd /mnt/media/Projects/portainer-core/stacks
 mv nextcloud.yml nextcloud-mariadb-archived.yml
 mv nextcloud-shared.yml nextcloud.yml
 # 4. Delete old archived stack (already purged data in Phase 1)
 # All old containers and data already removed
 ```
 ---
 ## Rollback Plan
 ⚠️ **NO ROLLBACK POSSIBLE** ⚠️
 Since all old data is purged in Phase 1, there is no rollback option.
 If fresh installation fails:
 1. Review error logs
 2. Fix configuration issues
 3. Retry fresh installation
 This is acceptable since Nextcloud is not in production use.
 ---
 ## Testing Checklist
 After fresh installation, verify:
 - [ ] Admin login works
 - [ ] File upload works
 - [ ] File download works
 - [ ] File delete works
 - [ ] Redis caching active (`docker exec redis-shared redis-cli -n 3 DBSIZE` shows keys)
 - [ ] PostgreSQL connection stable (`docker exec postgres-shared psql -U nextcloud_user -d nextcloud -c '\dt'` shows tables)
 - [ ] Memory usage acceptable (<1GB for Nextcloud container)
 - [ ] Nextcloud accessible via https://cloud.schweitz.net
 - [ ] No errors in logs (`docker logs nextcloud`)
 ---
 ## Resource Savings
 **Before Migration:**
 - nextcloud-db (MariaDB): ~117 MB RAM
 - nextcloud-redis: ~10 MB RAM
 - **Total:** ~127 MB RAM + 2 containers
 **After Migration:**
 - Shared postgres-shared: Already running (minimal additional overhead for one more DB)
 - Shared redis-shared: Already running (DB 3 uses ~5-10 MB additional)
 - **Savings:** ~110-120 MB RAM + 2 fewer containers to manage
 **Benefits:**
 - Simplified infrastructure
 - Centralized backups
 - Better resource utilization
 - Easier monitoring
 - Consistent database management
 ---
 ## Risks & Mitigation
 | Risk | Impact | Mitigation |
 |------|--------|------------|
 | Data loss during migration | HIGH | Full backups before starting, test on copy first |
 | Incompatible plugins/apps | MEDIUM | Fresh install allows clean app selection |
 | User resistance to re-setup | LOW | Minimal - same interface, same files |
 | Extended downtime | MEDIUM | Plan migration during low-usage window |
 | Redis DB conflict | LOW | Using dedicated DB 3, isolated from other apps |
 ---
 ## Timeline
 **Total estimated time:** 20-30 minutes
 - Phase 1: Purge all data: 2 min
 - Phase 2: Prepare PostgreSQL/Redis: 5 min
 - Phase 3: Create stack config: 2 min
 - Phase 4: Create directories: 1 min
 - Phase 5: Deploy Nextcloud: 3 min
 - Phase 6: Initial setup & config: 10 min
 - Phase 7: Testing: 5 min
 - Phase 8: Documentation: 2 min
 **Can be done anytime** - No production impact, no backups needed
 ---
 ## Approval Required
 - [ ] Backup strategy approved
 - [ ] Fresh install approach approved
 - [ ] Downtime window approved
 - [ ] Testing checklist reviewed
 - [ ] Rollback plan understood
 - [ ] Ready to proceed
 ---
 ## Notes
 - **COMPLETE FRESH START** - All old data deleted
 - Clean database, optimal performance from day one
 - PostgreSQL generally faster than MariaDB for Nextcloud workloads
 - Redis DB 3 dedicated to Nextcloud (isolated from other apps)
 - No migration complexity - just a clean installation
 - Ready for production use immediately after setup
@@ -22,7 +22,7 @@ services:
      - MYSQL_USER=nextcloud
      - TZ=Europe/Amsterdam
    networks:
-      - nextcloud-network
+      - docker-dataplane
  nextcloud-redis:
    image: redis:alpine
@@ -31,7 +31,7 @@ services:
    environment:
      - TZ=Europe/Amsterdam
    networks:
-      - nextcloud-network
+      - docker-dataplane
  nextcloud:
    image: nextcloud:stable
@@ -56,11 +56,12 @@ services:
      - nextcloud-db
      - nextcloud-redis
    networks:
-      - nextcloud-network
+      - docker-dataplane
 networks:
-  nextcloud-network:
+  docker-dataplane:
-    driver: bridge
+    external: true
    name: docker-dataplane
 # ⚠️ SECURITY WARNING:
 # Change MYSQL_ROOT_PASSWORD and MYSQL_PASSWORD before deploying!
@@ -24,11 +24,21 @@ services:
      - NVIDIA_DRIVER_CAPABILITIES=all
    deploy:
      resources:
        limits:
          memory: 8G
        reservations:
          memory: 1G
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    networks:
      - docker-dataplane
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 # GPU Requirements:
 # - RTX 2080 Ti (11GB VRAM)
@@ -1,5 +1,3 @@
 version: '3.8'
 services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
@@ -46,13 +44,21 @@ services:
    volumes:
      - /home/jpmschweitzer/docker-data/open-webui:/app/backend/data
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
        reservations:
          memory: 512M
    networks:
-      - ai-dataplane
+      - docker-dataplane
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
 networks:
-  ai-dataplane:
+  docker-dataplane:
-    driver: bridge
+    external: true
-    name: ai-dataplane
+    name: docker-dataplane
@@ -22,6 +22,13 @@ services:
      - PGID=1000
      - TZ=Europe/Amsterdam
      - fpm=true  # Enable PHP-FPM for better performance
    networks:
      - docker-dataplane
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 # Setup Instructions:
 # 1. Ensure tower-of-joy is connected to Headscale mesh (get mesh IP)
@@ -0,0 +1,135 @@
 version: '3.8'
 # Shared PostgreSQL Database
 # Purpose: Centralized database for all homelab applications
 # Port: 5432
 # GPU: No
 # Storage: SSD (PostgreSQL data and backups)
 services:
  postgres-shared:
    image: postgres:16-alpine
    container_name: postgres-shared
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 5s
    ports:
      - "5432:5432"
    volumes:
      - /home/jpmschweitzer/docker-data/postgres-shared/data:/var/lib/postgresql/data
      - /home/jpmschweitzer/docker-data/postgres-shared/backups:/backups
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_ADMIN_PASSWORD:?admin password required}
      TZ: Europe/Amsterdam
      # Performance tuning (adjust based on available RAM)
      # Shared buffers: 25% of RAM allocated to PostgreSQL
      POSTGRES_SHARED_BUFFERS: 512MB
      # Effective cache: 50-75% of RAM allocated to PostgreSQL
      POSTGRES_EFFECTIVE_CACHE_SIZE: 2GB
      # Max connections: adjust based on number of applications
      POSTGRES_MAX_CONNECTIONS: 200
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 2G
        reservations:
          memory: 512M
    networks:
      - docker-dataplane
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 # Setup Instructions:
 #
 # 1. Create directories:
 #    mkdir -p ~/docker-data/postgres-shared/{data,backups}
 #
 # 2. Deploy stack via core-api (recommended) or docker-compose
 #
 # 3. Initialize databases (run ONCE after first deployment):
 #    docker exec -i postgres-shared psql -U postgres <<'EOF'
 #      -- Authentik database
 #      CREATE DATABASE authentik;
 #      CREATE USER authentik_user WITH PASSWORD 'F//j0ktck7cX06Vfgh0YXceONOtlSsHvadqROICeDx8=';
 #      GRANT ALL PRIVILEGES ON DATABASE authentik TO authentik_user;
 #      \c authentik
 #      GRANT ALL ON SCHEMA public TO authentik_user;
 #      ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO authentik_user;
 #      ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO authentik_user;
 #
 #      -- Gitea database
 #      \c postgres
 #      CREATE DATABASE gitea;
 #      CREATE USER gitea_user WITH PASSWORD 'cCav64d76NX1zdEEAbVOM9uvao14aY8HojjNdxsSpMM=';
 #      GRANT ALL PRIVILEGES ON DATABASE gitea TO gitea_user;
 #      \c gitea
 #      GRANT ALL ON SCHEMA public TO gitea_user;
 #      ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO gitea_user;
 #      ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO gitea_user;
 #    EOF
 #
 # 4. Verify deployment:
 #    docker exec postgres-shared pg_isready
 #    docker exec postgres-shared psql -U postgres -c '\l'
 #
 # Database Connection Examples:
 #
 # From containers on docker-dataplane network:
 #   Host: postgres-shared
 #   Port: 5432
 #   Database: authentik (or gitea, etc.)
 #   User: authentik_user (or gitea_user, etc.)
 #   Password: <app-specific-password>
 #
 # From host machine:
 #   psql -h localhost -U authentik_user -d authentik
 #
 # Monitoring:
 #
 # Active connections per database:
 #   docker exec postgres-shared psql -U postgres -c \
 #     "SELECT datname, numbackends FROM pg_stat_database;"
 #
 # Database sizes:
 #   docker exec postgres-shared psql -U postgres -c \
 #     "SELECT datname, pg_size_pretty(pg_database_size(datname)) FROM pg_database;"
 #
 # Backup:
 #
 # All databases:
 #   docker exec postgres-shared pg_dumpall -U postgres | \
 #     gzip > ~/docker-data/postgres-shared/backups/all-$(date +%Y%m%d).sql.gz
 #
 # Single database:
 #   docker exec postgres-shared pg_dump -U postgres authentik | \
 #     gzip > ~/docker-data/postgres-shared/backups/authentik-$(date +%Y%m%d).sql.gz
 #
 # Restore:
 #   gunzip < backup.sql.gz | docker exec -i postgres-shared psql -U postgres
 #
 # Maintenance:
 #
 # Vacuum analyze (optimize performance):
 #   docker exec postgres-shared psql -U postgres -c "VACUUM ANALYZE;"
 #
 # Reindex (if queries slow):
 #   docker exec postgres-shared psql -U postgres -d authentik -c "REINDEX DATABASE authentik;"
 #
 # Resource Usage (expected):
 #   CPU: ~0.5-1.5 cores (depends on query load)
 #   RAM: ~500MB-1.5GB (depends on active connections and cache)
 #   Storage: Grows with data (monitor with: df -h ~/docker-data/postgres-shared)
 #
 # Applications Using This Database:
 #   - Authentik (identity provider)
 #   - Gitea (git hosting) - migrated from dedicated instance
 #   - Future applications as needed
@@ -21,12 +21,20 @@ services:
      - /home/jpmschweitzer/docker-data/qdrant/snapshots:/qdrant/snapshots
    environment:
      - TZ=Europe/Amsterdam
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 768M
        reservations:
          memory: 256M
    networks:
-      - ai-dataplane
+      - docker-dataplane
 networks:
-  ai-dataplane:
+  docker-dataplane:
    external: true
    name: docker-dataplane
 # Qdrant Performance Notes:
 # - Optimized for high-dimensional vectors (embeddings)
@@ -0,0 +1,153 @@
 version: '3.8'
 # Shared Redis Cache
 # Purpose: Centralized cache and session store for all homelab applications
 # Port: 6379
 # GPU: No
 # Storage: SSD (Redis persistence - AOF and RDB)
 services:
  redis-shared:
    image: redis:alpine
    container_name: redis-shared
    restart: unless-stopped
    command: >
      redis-server
      --appendonly yes
      --appendfsync everysec
      --maxmemory 512mb
      --maxmemory-policy allkeys-lru
      --save 60 1000
      --save 300 100
      --save 900 1
      --loglevel warning
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 3s
    ports:
      - "6379:6379"
    volumes:
      - /home/jpmschweitzer/docker-data/redis-shared/data:/data
    environment:
      TZ: Europe/Amsterdam
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          memory: 128M
    networks:
      - docker-dataplane
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 # Setup Instructions:
 #
 # 1. Create directories:
 #    mkdir -p ~/docker-data/redis-shared/data
 #
 # 2. Deploy stack:
 #    docker-compose -f redis-shared.yml up -d
 #
 # 3. Verify deployment:
 #    docker exec redis-shared redis-cli ping
 #
 # Database Allocation:
 #
 # Redis supports 16 databases (0-15). Assign one per application:
 #
 #   DB 0: General cache (default, shared lightweight caching)
 #   DB 1: Authentik (sessions, cache, message queue)
 #   DB 2: Gitea (cache, sessions)
 #   DB 3: Open WebUI (cache, if needed)
 #   DB 4-15: Reserved for future applications
 #
 # Connection Examples:
 #
 # From containers on docker-dataplane network:
 #   redis://redis-shared:6379/1  (Authentik, DB 1)
 #   redis://redis-shared:6379/2  (Gitea, DB 2)
 #
 # From host machine:
 #   redis-cli -h localhost
 #   SELECT 1  (switch to database 1)
 #
 # Monitoring:
 #
 # General info:
 #   docker exec redis-shared redis-cli INFO
 #
 # Memory usage:
 #   docker exec redis-shared redis-cli INFO memory
 #
 # Keyspace (keys per database):
 #   docker exec redis-shared redis-cli INFO keyspace
 #
 # Stats:
 #   docker exec redis-shared redis-cli INFO stats
 #
 # Per-database keys:
 #   docker exec redis-shared redis-cli -n 1 DBSIZE  (database 1)
 #   docker exec redis-shared redis-cli -n 2 DBSIZE  (database 2)
 #
 # Backup:
 #
 # Trigger background save:
 #   docker exec redis-shared redis-cli BGSAVE
 #
 # Copy RDB file:
 #   cp ~/docker-data/redis-shared/data/dump.rdb \
 #      ~/backups/redis-$(date +%Y%m%d).rdb
 #
 # Backup AOF (append-only file):
 #   cp ~/docker-data/redis-shared/data/appendonly.aof \
 #      ~/backups/redis-aof-$(date +%Y%m%d).aof
 #
 # Restore:
 #   docker stop redis-shared
 #   cp backup-dump.rdb ~/docker-data/redis-shared/data/dump.rdb
 #   docker start redis-shared
 #
 # Maintenance:
 #
 # Clear specific database (DANGER - data loss!):
 #   docker exec redis-shared redis-cli -n 1 FLUSHDB
 #
 # Clear all databases (DANGER - total data loss!):
 #   docker exec redis-shared redis-cli FLUSHALL
 #
 # Rewrite AOF (compact log file):
 #   docker exec redis-shared redis-cli BGREWRITEAOF
 #
 # Configuration Details:
 #
 # Persistence strategy (dual):
 #   - AOF (Append Only File): Real-time durability, fsync every second
 #   - RDB Snapshots: Periodic snapshots (every 60s if 1000+ keys changed)
 #
 # Memory policy:
 #   - Max memory: 512MB
 #   - Eviction: allkeys-lru (Least Recently Used eviction when full)
 #
 # Resource Usage (expected):
 #   CPU: ~0.1-0.3 cores (low CPU, very efficient)
 #   RAM: ~100-400MB (depends on data, capped at 512MB)
 #   Storage: ~50-200MB (AOF + RDB files)
 #
 # Applications Using This Cache:
 #   - Authentik (sessions, policies, background tasks)
 #   - Gitea (sessions, cache, queues) - if migrated
 #   - Future applications as needed
 #
 # Performance Tips:
 #   - Use pipeline commands for bulk operations
 #   - Set appropriate TTL (Time To Live) on cached keys
 #   - Monitor memory usage to prevent eviction storms
 #   - Use database numbers to isolate application data
@@ -32,6 +32,13 @@ services:
      -s "Backups;/share/backups;yes;no;yes;all"
      -u "jpmschweitzer;IG3omTybtVW3pVmmBi1D5FjnQ0MnZLUG"
      -p
    networks:
      - docker-dataplane
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 # ⚠️ SECURITY WARNING:
 # Change CHANGEME_SAMBA_PASSWORD before deploying!
@@ -18,25 +18,12 @@ services:
    environment:
      - TZ=Europe/Amsterdam
    networks:
-      - default
+      - docker-dataplane
      - ai-dataplane
      - nextcloud-network
      - headscale-network
      - samba-network
 networks:
-  ai-dataplane:
+  docker-dataplane:
    external: true
-    name: ai-dataplane
+    name: docker-dataplane
  nextcloud-network:
    external: true
    name: nextcloud_nextcloud-network
  headscale-network:
    external: true
    name: stacks_headscale-network
  samba-network:
    external: true
    name: samba_default
 # After Deployment:
 # 1. Access http://localhost:3001
@@ -23,6 +23,13 @@ services:
      # Optional: Monitor only specific containers
      # - WATCHTOWER_LABEL_ENABLE=true  # Only update containers with label com.centurylinklabs.watchtower.enable=true
    networks:
      - docker-dataplane
 networks:
  docker-dataplane:
    external: true
    name: docker-dataplane
 # Schedule Format (cron):
 # - 0 0 4 * * * = Daily at 4 AM
@@ -0,0 +1,62 @@
 <!DOCTYPE html>
 <html>
 <head>
    <title>Header Test</title>
    <style>
        body { font-family: monospace; padding: 20px; background: #1a1a1a; color: #00ff00; }
        h1 { color: #00ff00; }
        .header { padding: 5px; margin: 2px 0; background: #2a2a2a; }
        .header-name { color: #00aaff; font-weight: bold; }
        .header-value { color: #ffff00; }
        .good { background: #003300; }
    </style>
 </head>
 <body>
    <h1>Authentik Header Test</h1>
    <p>This page will request headers from your current session</p>
    <div id="results">Loading...</div>
    <script>
        // Make a request to a simple echo endpoint
        fetch('/api/test-headers')
            .then(r => r.json())
            .then(data => {
                const headers = data.headers || {};
                let html = '<h2>Received Headers:</h2>';
                const authentikHeaders = [
                    'x-authentik-username',
                    'x-authentik-email',
                    'x-authentik-groups',
                    'x-authentik-name',
                    'x-authentik-uid'
                ];
                for (let header of authentikHeaders) {
                    const value = headers[header] || headers[header.toUpperCase()] || '(not found)';
                    const hasValue = value !== '(not found)';
                    html += `<div class="header ${hasValue ? 'good' : ''}">`;
                    html += `<span class="header-name">${header}:</span> `;
                    html += `<span class="header-value">${value}</span>`;
                    html += `</div>`;
                }
                html += '<h2>All Headers:</h2>';
                for (let [key, value] of Object.entries(headers)) {
                    html += `<div class="header">`;
                    html += `<span class="header-name">${key}:</span> `;
                    html += `<span class="header-value">${value}</span>`;
                    html += `</div>`;
                }
                document.getElementById('results').innerHTML = html;
            })
            .catch(e => {
                document.getElementById('results').innerHTML =
                    `<p style="color: red;">Error: ${e.message}</p>
                     <p>Your Organizr installation may not have an API endpoint we can test with.</p>
                     <p>Please check the Organizr logs or settings to see what headers it's receiving.</p>`;
            });
    </script>
 </body>
 </html>