portainer-core/docs/backup-procedures.md

Backup & Restore Procedures

Overview

The maintenance container runs scheduled backup tasks using cron. It's a simple, reliable, "set and forget" solution.

Current Backups:

• Docker Configs: Daily at 3 AM
• Retention: 30 days
• Size: ~94 MB per backup
• Location: /mnt/media/backups/docker-configs/

What's Backed Up:

• ✅ All Docker container configurations
• ✅ Nginx Proxy Manager configs & SSL certificates
• ✅ Headscale database & config
• ✅ All dashboard settings (Heimdall, Organizr, Uptime Kuma)
• ✅ All service configs
• ❌ Ollama models (re-downloadable)
• ❌ Cache files
• ❌ Log files

Automated Backups

Schedule: Daily at 3:00 AM (configured in crontab)

View Backup Logs:

# Real-time logs
docker logs -f maintenance

# Backup script logs
cat ~/docker-data/maintenance/logs/backup-configs.log

List Existing Backups:

ls -lh /mnt/media/backups/docker-configs/

Manual Backup

Run a backup anytime:

docker exec maintenance /scripts/backup-configs.sh

Restore from Backup

Full Restore

1. Stop all containers:

   docker stop $(docker ps -aq)
   

2. Backup current state (just in case):

   mv ~/docker-data ~/docker-data.old
   

3. Extract backup:

   cd ~
   tar -xzf /mnt/media/backups/docker-configs/docker-configs-YYYYMMDD-HHMMSS.tar.gz
   

4. Restart containers:

   docker start $(docker ps -aq)
   

5. Verify services:

   docker ps
   

Selective Restore (Single Service)

Restore only one service's config (example: Headscale):

# Extract only headscale directory
tar -xzf /mnt/media/backups/docker-configs/docker-configs-20251111-221349.tar.gz \
    --strip-components=2 \
    -C ~/docker-data/ \
    docker-data/headscale

# Restart the service
docker restart headscale

Adding New Maintenance Tasks

The maintenance container can run any scheduled task, not just backups.

1. Create New Script

# Create script file
nano ~/docker-data/maintenance/scripts/my-task.sh

# Make it executable
chmod +x ~/docker-data/maintenance/scripts/my-task.sh

2. Add to Crontab

# Edit crontab
nano ~/docker-data/maintenance/crontab

# Add your schedule (example: every Sunday at 4 AM)
# 0 4 * * 0 /scripts/my-task.sh

3. Restart Container

docker restart maintenance

Examples of Future Tasks

• Weekly cleanup: Remove old Docker images
• Health checks: Verify all services are responding
• Update checks: Notify when container updates available
• Database optimization: Compact/optimize databases
• SSL renewal checks: Verify certificates are valid

Testing Backup Integrity

Periodically test that backups can be restored:

# Create test directory
mkdir -p /tmp/backup-test

# Extract backup
tar -xzf /mnt/media/backups/docker-configs/docker-configs-LATEST.tar.gz \
    -C /tmp/backup-test

# Verify contents
ls -la /tmp/backup-test/docker-data/

# Clean up
rm -rf /tmp/backup-test

Troubleshooting

Backup Not Running

Check if container is running:

docker ps | grep maintenance

Check cron logs:

docker logs maintenance

Manually run backup to test:

docker exec maintenance /scripts/backup-configs.sh

Backup Taking Too Long

• Check if exclusions are working (Ollama models should be excluded)
• Monitor disk I/O: iostat -x 1
• Check HDD health: sudo smartctl -a /dev/sdb

Backup Disk Full

• Old backups auto-delete after 30 days
• Manually remove old backups if needed:

  # List backups by size
  du -h /mnt/media/backups/docker-configs/*
  
  # Remove specific backup
  rm /mnt/media/backups/docker-configs/docker-configs-20251001-*.tar.gz
  

Restore Failed

1. Check backup file integrity:

   tar -tzf /mnt/media/backups/docker-configs/backup-file.tar.gz > /dev/null
   

2. If corrupted, try previous backup

3. Check disk space before restoring:

   df -h ~/docker-data
   

Backup Storage

Current Usage:

• ~94 MB per daily backup
• 30 days retention = ~2.8 GB total
• Stored on 3.6 TB HDD (plenty of space)

Offsite Backups (Recommended):

For extra protection, periodically copy backups to external drive:

# Copy last 7 days to external drive
rsync -av --progress /mnt/media/backups/docker-configs/ /mnt/external-drive/backups/

---

Last Updated: 2025-11-11