Files

T

jpmschweitzer 0e3fef20fc restructure documentation

2025-11-20 10:17:09 +01:00

10 KiB

Raw Blame History

Shared Infrastructure Architecture

Purpose: Centralized PostgreSQL and Redis services for all homelab stacks Benefits: Resource efficiency, easier maintenance, unified backups, centralized monitoring

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                  Application Stacks                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
│  │Authentik │  │  Gitea   │  │ Future   │  │ Future   │   │
│  │          │  │          │  │  Stack   │  │  Stack   │   │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘   │
│       │             │              │              │          │
└───────┼─────────────┼──────────────┼──────────────┼─────────┘
        │             │              │              │
        └─────────────┴──────────────┴──────────────┘
                      │
        ┌─────────────▼──────────────────────────────┐
        │      Unified Data Plane Network            │
        │          (docker-dataplane)                │
        └─────────────┬──────────────────────────────┘
                      │
        ┌─────────────┴──────────────┐
        │                            │
   ┌────▼──────┐            ┌────────▼────┐
   │PostgreSQL │            │    Redis    │
   │  Shared   │            │   Shared    │
   │           │            │             │
   │ Databases:│            │ DB 0: Cache │
   │  - auth   │            │ DB 1: Auth  │
   │  - gitea  │            │ DB 2: Gitea │
   │  - future │            │ DB 3-15: .. │
   └───────────┘            └─────────────┘

Design Principles

1. Database Isolation

Each application gets its own PostgreSQL database within the shared instance
Each application gets its own Redis database number (0-15)
Separate credentials per application for security

2. Network Architecture

Unified network: docker-dataplane (external, bridge)
All application containers connect to this single network
Simplified connectivity: services discover each other by container name
Replaces per-stack networks (ai-dataplane, nextcloud-network, etc.)

3. Resource Allocation

PostgreSQL: No hard limits (homelab resource availability)
Redis: No hard limits (lightweight Alpine image)
Shared instances more efficient than per-stack deployments

4. Backup Strategy

Single PostgreSQL backup covers all databases
Automated pg_dumpall for disaster recovery
Redis persistence: AOF + RDB snapshots

5. Security Model

Each app has dedicated PostgreSQL user with access only to its database
Redis AUTH with per-database passwords (optional)
Network-level isolation via Docker networks

Database Allocation Plan

PostgreSQL Databases

Database Name	Application	User	Purpose
`authentik`	Authentik	`authentik_user`	User/group/policy storage
`gitea`	Gitea	`gitea_user`	Git repos, users, issues
`future_app1`	TBD	`app1_user`	Reserved
`future_app2`	TBD	`app2_user`	Reserved

Note: Existing services stay as-is:

Nextcloud: MariaDB (existing, not migrated)
Others can migrate over time if beneficial

Redis Database Numbers

DB#	Application	Purpose
0	Authentik	Sessions, cache, message queue
1	Available	Reserved for future applications
2	Available	Reserved for future applications
3-15	Available	Reserved for future applications

Note: Each application uses a dedicated DB number to prevent key collisions while sharing the same Redis instance.

Connection Configuration

PostgreSQL Connection Strings

From Docker containers:

Host: postgres-shared
Port: 5432
Database: authentik
User: authentik_user
Password: <app-specific-password>

From host:

Host: localhost
Port: 5432
Database: authentik
User: authentik_user
Password: <app-specific-password>

Redis Connection Strings

From Docker containers:

redis://redis-shared:6379/0  (for Authentik, DB 0)
redis://redis-shared:6379/1  (for future apps, DB 1)
redis://redis-shared:6379/2  (for future apps, DB 2)

From host:

redis://localhost:6379/0

Migration Strategy

Phase 1: Deploy Shared Infrastructure ✅ COMPLETE

✅ Deployed postgres-shared.yml and redis-shared.yml via Portainer
✅ Verified PostgreSQL 17 and Redis 7 running on docker-dataplane
✅ Created initial databases and users (authentik, gitea)
✅ Both services monitored via Uptime Kuma

Phase 2: New Services (Authentik) 🚧 IN PROGRESS

⏳ Deploy Authentik pointing to shared services
⏳ Test thoroughly
⏳ Validate no performance degradation

Phase 3: Network Consolidation ✅ COMPLETE

✅ All services migrated to docker-dataplane network
✅ Removed 7 obsolete Docker networks
✅ 18 containers on unified network for service discovery

Phase 4: Migrate Existing Services (Optional)

Gitea: Already uses PostgreSQL
- Export existing database
- Create gitea database in shared PostgreSQL
- Import data
- Update Gitea stack to use shared PostgreSQL
- Remove old gitea-db container
Other services: Evaluate case-by-case
- Nextcloud: Keep MariaDB (complex migration, low benefit)
- Future services: Use shared from day 1

Advantages

✅ Resource Efficiency

One PostgreSQL instance: ~1GB RAM vs ~300MB per instance
Saves ~700MB RAM per additional service using PostgreSQL

✅ Operational Simplicity

Single backup process for all PostgreSQL databases
Centralized monitoring and health checks
Easier version upgrades (upgrade once, affects all)

✅ Performance

Shared connection pooling
Better resource utilization
Optimized caching with shared Redis

✅ Scalability

Add new applications without deploying new database instances
Up to 15 Redis databases (more than enough for homelab)

Disadvantages & Mitigations

⚠️ Single Point of Failure

Mitigation: Health checks, automated restarts, regular backups
Acceptable for homelab: VPN access ensures admin can fix issues

⚠️ Resource Contention

Mitigation: PostgreSQL connection limits per database
Mitigation: Redis max memory policy (LRU eviction)
Monitoring: Track per-database usage

⚠️ Version Lock-In

Mitigation: Use latest stable PostgreSQL version (17)
Mitigation: Test upgrades in staging before production deployment

Monitoring & Maintenance

Health Checks

PostgreSQL: pg_isready every 30s
Redis: redis-cli ping every 30s
Application connectivity tests

Uptime Kuma Integration ✅ DEPLOYED

Both shared services are monitored via Uptime Kuma with automatic monitor creation through the Core API:

PostgreSQL Monitor (ID 20):

curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
  -H "Content-Type: application/json" \
  -d '{
    "type": "postgres",
    "name": "PostgreSQL Shared",
    "interval": 60,
    "retryInterval": 60,
    "maxretries": 3,
    "notificationIDList": [],
    "accepted_statuscodes": ["200-299"],
    "databaseConnectionString": "postgres://postgres:<url-encoded-password>@postgres-shared:5432/postgres"
  }'

Redis Monitor (ID 18):

curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
  -H "Content-Type: application/json" \
  -d '{
    "type": "port",
    "name": "Redis Shared - Port Check",
    "hostname": "redis-shared",
    "port": 6379,
    "interval": 60,
    "retryInterval": 60,
    "maxretries": 3,
    "notificationIDList": [],
    "accepted_statuscodes": ["200-299"]
  }'

Note: When monitoring PostgreSQL with passwords containing special characters, URL-encode them (/ → %2F, = → %3D).

Backup Schedule

PostgreSQL: Manual pg_dump to /backups/ volume (automated backups pending)
Redis: AOF persistence (real-time) enabled via --appendonly yes

Performance Monitoring

Query: SELECT datname, numbackends FROM pg_stat_database; (active connections)
Redis: INFO stats (keyspace usage per database)
Uptime Kuma dashboard: Real-time availability tracking

Upgrade Path

Backup all databases
Test upgrade with docker-compose override
Deploy new version
Verify all applications connect successfully
Rollback if issues detected

Implementation Status

✅ Review architecture design
✅ Create postgres-shared.yml and redis-shared.yml stacks
✅ Deploy shared PostgreSQL 17 and Redis 7 via Portainer
✅ Create initial databases (authentik, gitea)
✅ Consolidate all services to docker-dataplane network
✅ Implement Uptime Kuma monitoring via Core API
✅ Document connection patterns and deployment procedures
⏳ Update authentik.yml to use shared services (pending)
⏳ Test Authentik with shared infrastructure (pending)

Future Enhancements

PostgreSQL Read Replicas (if needed for heavy read workloads)
Redis Sentinel (high availability, probably overkill for homelab)
PgBouncer (connection pooling if >100 connections needed)
Prometheus + Grafana (metrics visualization)

10 KiB Raw Blame History