restructure documentation
This commit is contained in:
@@ -0,0 +1,292 @@
|
||||
# Shared Infrastructure Architecture
|
||||
|
||||
**Purpose:** Centralized PostgreSQL and Redis services for all homelab stacks
|
||||
**Benefits:** Resource efficiency, easier maintenance, unified backups, centralized monitoring
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Application Stacks │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │Authentik │ │ Gitea │ │ Future │ │ Future │ │
|
||||
│ │ │ │ │ │ Stack │ │ Stack │ │
|
||||
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
|
||||
│ │ │ │ │ │
|
||||
└───────┼─────────────┼──────────────┼──────────────┼─────────┘
|
||||
│ │ │ │
|
||||
└─────────────┴──────────────┴──────────────┘
|
||||
│
|
||||
┌─────────────▼──────────────────────────────┐
|
||||
│ Unified Data Plane Network │
|
||||
│ (docker-dataplane) │
|
||||
└─────────────┬──────────────────────────────┘
|
||||
│
|
||||
┌─────────────┴──────────────┐
|
||||
│ │
|
||||
┌────▼──────┐ ┌────────▼────┐
|
||||
│PostgreSQL │ │ Redis │
|
||||
│ Shared │ │ Shared │
|
||||
│ │ │ │
|
||||
│ Databases:│ │ DB 0: Cache │
|
||||
│ - auth │ │ DB 1: Auth │
|
||||
│ - gitea │ │ DB 2: Gitea │
|
||||
│ - future │ │ DB 3-15: .. │
|
||||
└───────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Design Principles
|
||||
|
||||
### 1. **Database Isolation**
|
||||
- Each application gets its own PostgreSQL database within the shared instance
|
||||
- Each application gets its own Redis database number (0-15)
|
||||
- Separate credentials per application for security
|
||||
|
||||
### 2. **Network Architecture**
|
||||
- **Unified network:** `docker-dataplane` (external, bridge)
|
||||
- All application containers connect to this single network
|
||||
- Simplified connectivity: services discover each other by container name
|
||||
- Replaces per-stack networks (ai-dataplane, nextcloud-network, etc.)
|
||||
|
||||
### 3. **Resource Allocation**
|
||||
- PostgreSQL: No hard limits (homelab resource availability)
|
||||
- Redis: No hard limits (lightweight Alpine image)
|
||||
- Shared instances more efficient than per-stack deployments
|
||||
|
||||
### 4. **Backup Strategy**
|
||||
- Single PostgreSQL backup covers all databases
|
||||
- Automated pg_dumpall for disaster recovery
|
||||
- Redis persistence: AOF + RDB snapshots
|
||||
|
||||
### 5. **Security Model**
|
||||
- Each app has dedicated PostgreSQL user with access only to its database
|
||||
- Redis AUTH with per-database passwords (optional)
|
||||
- Network-level isolation via Docker networks
|
||||
|
||||
---
|
||||
|
||||
## Database Allocation Plan
|
||||
|
||||
### PostgreSQL Databases
|
||||
|
||||
| Database Name | Application | User | Purpose |
|
||||
|---------------|-------------|------|---------|
|
||||
| `authentik` | Authentik | `authentik_user` | User/group/policy storage |
|
||||
| `gitea` | Gitea | `gitea_user` | Git repos, users, issues |
|
||||
| `future_app1` | TBD | `app1_user` | Reserved |
|
||||
| `future_app2` | TBD | `app2_user` | Reserved |
|
||||
|
||||
**Note:** Existing services stay as-is:
|
||||
- Nextcloud: MariaDB (existing, not migrated)
|
||||
- Others can migrate over time if beneficial
|
||||
|
||||
### Redis Database Numbers
|
||||
|
||||
| DB# | Application | Purpose |
|
||||
|-----|-------------|---------|
|
||||
| 0 | Authentik | Sessions, cache, message queue |
|
||||
| 1 | Available | Reserved for future applications |
|
||||
| 2 | Available | Reserved for future applications |
|
||||
| 3-15 | Available | Reserved for future applications |
|
||||
|
||||
**Note:** Each application uses a dedicated DB number to prevent key collisions while sharing the same Redis instance.
|
||||
|
||||
---
|
||||
|
||||
## Connection Configuration
|
||||
|
||||
### PostgreSQL Connection Strings
|
||||
|
||||
**From Docker containers:**
|
||||
```
|
||||
Host: postgres-shared
|
||||
Port: 5432
|
||||
Database: authentik
|
||||
User: authentik_user
|
||||
Password: <app-specific-password>
|
||||
```
|
||||
|
||||
**From host:**
|
||||
```
|
||||
Host: localhost
|
||||
Port: 5432
|
||||
Database: authentik
|
||||
User: authentik_user
|
||||
Password: <app-specific-password>
|
||||
```
|
||||
|
||||
### Redis Connection Strings
|
||||
|
||||
**From Docker containers:**
|
||||
```
|
||||
redis://redis-shared:6379/0 (for Authentik, DB 0)
|
||||
redis://redis-shared:6379/1 (for future apps, DB 1)
|
||||
redis://redis-shared:6379/2 (for future apps, DB 2)
|
||||
```
|
||||
|
||||
**From host:**
|
||||
```
|
||||
redis://localhost:6379/0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Phase 1: Deploy Shared Infrastructure ✅ **COMPLETE**
|
||||
1. ✅ Deployed `postgres-shared.yml` and `redis-shared.yml` via Portainer
|
||||
2. ✅ Verified PostgreSQL 17 and Redis 7 running on docker-dataplane
|
||||
3. ✅ Created initial databases and users (authentik, gitea)
|
||||
4. ✅ Both services monitored via Uptime Kuma
|
||||
|
||||
### Phase 2: New Services (Authentik) 🚧 **IN PROGRESS**
|
||||
1. ⏳ Deploy Authentik pointing to shared services
|
||||
2. ⏳ Test thoroughly
|
||||
3. ⏳ Validate no performance degradation
|
||||
|
||||
### Phase 3: Network Consolidation ✅ **COMPLETE**
|
||||
1. ✅ All services migrated to docker-dataplane network
|
||||
2. ✅ Removed 7 obsolete Docker networks
|
||||
3. ✅ 18 containers on unified network for service discovery
|
||||
|
||||
### Phase 4: Migrate Existing Services (Optional)
|
||||
1. **Gitea**: Already uses PostgreSQL
|
||||
- Export existing database
|
||||
- Create gitea database in shared PostgreSQL
|
||||
- Import data
|
||||
- Update Gitea stack to use shared PostgreSQL
|
||||
- Remove old gitea-db container
|
||||
|
||||
2. **Other services**: Evaluate case-by-case
|
||||
- Nextcloud: Keep MariaDB (complex migration, low benefit)
|
||||
- Future services: Use shared from day 1
|
||||
|
||||
---
|
||||
|
||||
## Advantages
|
||||
|
||||
✅ **Resource Efficiency**
|
||||
- One PostgreSQL instance: ~1GB RAM vs ~300MB per instance
|
||||
- Saves ~700MB RAM per additional service using PostgreSQL
|
||||
|
||||
✅ **Operational Simplicity**
|
||||
- Single backup process for all PostgreSQL databases
|
||||
- Centralized monitoring and health checks
|
||||
- Easier version upgrades (upgrade once, affects all)
|
||||
|
||||
✅ **Performance**
|
||||
- Shared connection pooling
|
||||
- Better resource utilization
|
||||
- Optimized caching with shared Redis
|
||||
|
||||
✅ **Scalability**
|
||||
- Add new applications without deploying new database instances
|
||||
- Up to 15 Redis databases (more than enough for homelab)
|
||||
|
||||
---
|
||||
|
||||
## Disadvantages & Mitigations
|
||||
|
||||
⚠️ **Single Point of Failure**
|
||||
- **Mitigation:** Health checks, automated restarts, regular backups
|
||||
- **Acceptable for homelab:** VPN access ensures admin can fix issues
|
||||
|
||||
⚠️ **Resource Contention**
|
||||
- **Mitigation:** PostgreSQL connection limits per database
|
||||
- **Mitigation:** Redis max memory policy (LRU eviction)
|
||||
- **Monitoring:** Track per-database usage
|
||||
|
||||
⚠️ **Version Lock-In**
|
||||
- **Mitigation:** Use latest stable PostgreSQL version (17)
|
||||
- **Mitigation:** Test upgrades in staging before production deployment
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Maintenance
|
||||
|
||||
### Health Checks
|
||||
- PostgreSQL: `pg_isready` every 30s
|
||||
- Redis: `redis-cli ping` every 30s
|
||||
- Application connectivity tests
|
||||
|
||||
### Uptime Kuma Integration ✅ **DEPLOYED**
|
||||
|
||||
Both shared services are monitored via Uptime Kuma with automatic monitor creation through the Core API:
|
||||
|
||||
**PostgreSQL Monitor** (ID 20):
|
||||
```bash
|
||||
curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"type": "postgres",
|
||||
"name": "PostgreSQL Shared",
|
||||
"interval": 60,
|
||||
"retryInterval": 60,
|
||||
"maxretries": 3,
|
||||
"notificationIDList": [],
|
||||
"accepted_statuscodes": ["200-299"],
|
||||
"databaseConnectionString": "postgres://postgres:<url-encoded-password>@postgres-shared:5432/postgres"
|
||||
}'
|
||||
```
|
||||
|
||||
**Redis Monitor** (ID 18):
|
||||
```bash
|
||||
curl -X POST http://192.168.86.149:8083/infrastructure/monitors \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"type": "port",
|
||||
"name": "Redis Shared - Port Check",
|
||||
"hostname": "redis-shared",
|
||||
"port": 6379,
|
||||
"interval": 60,
|
||||
"retryInterval": 60,
|
||||
"maxretries": 3,
|
||||
"notificationIDList": [],
|
||||
"accepted_statuscodes": ["200-299"]
|
||||
}'
|
||||
```
|
||||
|
||||
**Note:** When monitoring PostgreSQL with passwords containing special characters, URL-encode them (`/` → `%2F`, `=` → `%3D`).
|
||||
|
||||
### Backup Schedule
|
||||
- **PostgreSQL:** Manual pg_dump to `/backups/` volume (automated backups pending)
|
||||
- **Redis:** AOF persistence (real-time) enabled via `--appendonly yes`
|
||||
|
||||
### Performance Monitoring
|
||||
- Query: `SELECT datname, numbackends FROM pg_stat_database;` (active connections)
|
||||
- Redis: `INFO stats` (keyspace usage per database)
|
||||
- Uptime Kuma dashboard: Real-time availability tracking
|
||||
|
||||
### Upgrade Path
|
||||
1. Backup all databases
|
||||
2. Test upgrade with docker-compose override
|
||||
3. Deploy new version
|
||||
4. Verify all applications connect successfully
|
||||
5. Rollback if issues detected
|
||||
|
||||
---
|
||||
|
||||
## Implementation Status
|
||||
|
||||
1. ✅ Review architecture design
|
||||
2. ✅ Create `postgres-shared.yml` and `redis-shared.yml` stacks
|
||||
3. ✅ Deploy shared PostgreSQL 17 and Redis 7 via Portainer
|
||||
4. ✅ Create initial databases (authentik, gitea)
|
||||
5. ✅ Consolidate all services to docker-dataplane network
|
||||
6. ✅ Implement Uptime Kuma monitoring via Core API
|
||||
7. ✅ Document connection patterns and deployment procedures
|
||||
8. ⏳ Update `authentik.yml` to use shared services (pending)
|
||||
9. ⏳ Test Authentik with shared infrastructure (pending)
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **PostgreSQL Read Replicas** (if needed for heavy read workloads)
|
||||
- **Redis Sentinel** (high availability, probably overkill for homelab)
|
||||
- **PgBouncer** (connection pooling if >100 connections needed)
|
||||
- **Prometheus + Grafana** (metrics visualization)
|
||||
Reference in New Issue
Block a user