portainer-core/docs/architecture/SHARED_INFRASTRUCTURE_ARCHITECTURE.md

Shared Infrastructure Architecture

Purpose: Centralized PostgreSQL and Redis services for all homelab stacks Benefits: Resource efficiency, easier maintenance, unified backups, centralized monitoring

---

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                  Application Stacks                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
│  │Authentik │  │  Gitea   │  │ Organizr │  │ Future   │   │
│  │          │  │          │  │          │  │  Stack   │   │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘   │
│       │             │              │              │          │
└───────┼─────────────┼──────────────┼──────────────┼─────────┘
        │             │              │              │
        └─────────────┴──────────────┴──────────────┘
                      │
        ┌─────────────▼──────────────────────────────┐
        │      Unified Data Plane Network            │
        │          (docker-dataplane)                │
        └─────────────┬──────────────────────────────┘
                      │
        ┌─────────────┴──────────────┐
        │                            │
   ┌────▼──────┐            ┌────────▼────┐
   │PostgreSQL │            │    Redis    │
   │  Shared   │            │   Shared    │
   │           │            │             │
   │ Databases:│            │ DB 0: Cache │
   │  - auth   │            │ DB 1: Auth  │
   │  - gitea  │            │ DB 2: Gitea │
   │  - organizr│            │ DB 3-15: .. │
   │  - future │            │             │
   └───────────┘            └─────────────┘

---

Design Principles

1. Database Isolation

• Each application gets its own PostgreSQL database within the shared instance
• Each application gets its own Redis database number (0-15)
• Separate credentials per application for security

2. Network Architecture

• Unified network: docker-dataplane (external, bridge)
• All application containers connect to this single network
• Simplified connectivity: services discover each other by container name
• Replaces per-stack networks (ai-dataplane, nextcloud-network, etc.)

3. Resource Allocation

• PostgreSQL: No hard limits (homelab resource availability)
• Redis: No hard limits (lightweight Alpine image)
• Shared instances more efficient than per-stack deployments

4. Backup Strategy

• Single PostgreSQL backup covers all databases
• Automated pg_dumpall for disaster recovery
• Redis persistence: AOF + RDB snapshots

5. Security Model

• Each app has dedicated PostgreSQL user with access only to its database
• Redis AUTH with per-database passwords (optional)
• Network-level isolation via Docker networks

---

Database Allocation Plan

PostgreSQL Databases

Database Name	Application	User	Purpose
authentik	Authentik	authentik_user	User/group/policy storage
gitea	Gitea	gitea_user	Git repos, users, issues
organizr	Organizr	organizr_user	Dashboard configuration and user data
future_app1	TBD	app1_user	Reserved
future_app2	TBD	app2_user	Reserved

Note: Existing services stay as-is:

• Nextcloud: MariaDB (existing, not migrated)
• Others can migrate over time if beneficial

Redis Database Numbers

DB#	Application	Purpose
0	Authentik	Sessions, cache, message queue
1	Available	Reserved for future applications
2	Available	Reserved for future applications
3-15	Available	Reserved for future applications

Note: Each application uses a dedicated DB number to prevent key collisions while sharing the same Redis instance.

---

Connection Configuration

PostgreSQL Connection Strings

From Docker containers:

Host: postgres-shared
Port: 5432
Database: authentik
User: authentik_user
Password: <app-specific-password>

From host:

Host: localhost
Port: 5432
Database: authentik
User: authentik_user
Password: <app-specific-password>

Redis Connection Strings

From Docker containers:

redis://redis-shared:6379/0  (for Authentik, DB 0)
redis://redis-shared:6379/1  (for future apps, DB 1)
redis://redis-shared:6379/2  (for future apps, DB 2)

From host:

redis://localhost:6379/0

---

Migration Strategy

Phase 1: Deploy Shared Infrastructure ✅ COMPLETE

1. ✅ Deployed postgres-shared.yml and redis-shared.yml via Portainer
2. ✅ Verified PostgreSQL 17 and Redis 7 running on docker-dataplane
3. ✅ Created initial databases and users (authentik, gitea)
4. ✅ Both services monitored via Uptime Kuma

Phase 2: New Services (Authentik) 🚧 IN PROGRESS

1. ⏳ Deploy Authentik pointing to shared services
2. ⏳ Test thoroughly
3. ⏳ Validate no performance degradation

Phase 3: Network Consolidation ✅ COMPLETE

1. ✅ All services migrated to docker-dataplane network
2. ✅ Removed 7 obsolete Docker networks
3. ✅ 18 containers on unified network for service discovery

Phase 4: Migrate Existing Services (Optional)

1. Gitea: Already uses PostgreSQL

   • Export existing database
   • Create gitea database in shared PostgreSQL
   • Import data
   • Update Gitea stack to use shared PostgreSQL
   • Remove old gitea-db container

2. Other services: Evaluate case-by-case

   • Nextcloud: Keep MariaDB (complex migration, low benefit)
   • Future services: Use shared from day 1

---

Advantages

✅ Resource Efficiency

• One PostgreSQL instance: ~1GB RAM vs ~300MB per instance
• Saves ~700MB RAM per additional service using PostgreSQL

✅ Operational Simplicity

• Single backup process for all PostgreSQL databases
• Centralized monitoring and health checks
• Easier version upgrades (upgrade once, affects all)

✅ Performance

• Shared connection pooling
• Better resource utilization
• Optimized caching with shared Redis

✅ Scalability

• Add new applications without deploying new database instances
• Up to 15 Redis databases (more than enough for homelab)

---

Disadvantages & Mitigations

⚠️ Single Point of Failure

• Mitigation: Health checks, automated restarts, regular backups
• Acceptable for homelab: VPN access ensures admin can fix issues

⚠️ Resource Contention

• Mitigation: PostgreSQL connection limits per database
• Mitigation: Redis max memory policy (LRU eviction)
• Monitoring: Track per-database usage

⚠️ Version Lock-In

• Mitigation: Use latest stable PostgreSQL version (17)
• Mitigation: Test upgrades in staging before production deployment

---

Monitoring & Maintenance

Health Checks

• PostgreSQL: pg_isready every 30s
• Redis: redis-cli ping every 30s
• Application connectivity tests

Uptime Kuma Integration ✅ DEPLOYED

Both shared services are monitored via Uptime Kuma with automatic monitor creation through the Core API:

PostgreSQL Monitor (ID 20):

curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
  -H "Content-Type: application/json" \
  -d '{
    "type": "postgres",
    "name": "PostgreSQL Shared",
    "interval": 60,
    "retryInterval": 60,
    "maxretries": 3,
    "notificationIDList": [],
    "accepted_statuscodes": ["200-299"],
    "databaseConnectionString": "postgres://postgres:<url-encoded-password>@postgres-shared:5432/postgres"
  }'

Redis Monitor (ID 18):

curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
  -H "Content-Type: application/json" \
  -d '{
    "type": "port",
    "name": "Redis Shared - Port Check",
    "hostname": "redis-shared",
    "port": 6379,
    "interval": 60,
    "retryInterval": 60,
    "maxretries": 3,
    "notificationIDList": [],
    "accepted_statuscodes": ["200-299"]
  }'

Note: When monitoring PostgreSQL with passwords containing special characters, URL-encode them (/ → %2F, = → %3D).

Backup Schedule

• PostgreSQL: Manual pg_dump to /backups/ volume (automated backups pending)
• Redis: AOF persistence (real-time) enabled via --appendonly yes

Performance Monitoring

• Query: SELECT datname, numbackends FROM pg_stat_database; (active connections)
• Redis: INFO stats (keyspace usage per database)
• Uptime Kuma dashboard: Real-time availability tracking

Upgrade Path

1. Backup all databases
2. Test upgrade with docker-compose override
3. Deploy new version
4. Verify all applications connect successfully
5. Rollback if issues detected

---

Implementation Status

1. ✅ Review architecture design
2. ✅ Create postgres-shared.yml and redis-shared.yml stacks
3. ✅ Deploy shared PostgreSQL 17 and Redis 7 via Portainer
4. ✅ Create initial databases (authentik, gitea)
5. ✅ Consolidate all services to docker-dataplane network
6. ✅ Implement Uptime Kuma monitoring via Core API
7. ✅ Document connection patterns and deployment procedures
8. ⏳ Update authentik.yml to use shared services (pending)
9. ⏳ Test Authentik with shared infrastructure (pending)

---

Future Enhancements

• PostgreSQL Read Replicas (if needed for heavy read workloads)
• Redis Sentinel (high availability, probably overkill for homelab)
• PgBouncer (connection pooling if >100 connections needed)
• Prometheus + Grafana (metrics visualization)