portainer-core/docs/guides/backup-procedures.md

# Backup & Restore Procedures

## Overview

The `maintenance` container runs scheduled backup tasks using cron. It's a simple, reliable, "set and forget" solution.

**Current Backups:**
- **Docker Configs:** Daily at 3 AM
- **Retention:** 30 days
- **Size:** ~94 MB per backup
- **Location:** `/mnt/media/backups/docker-configs/`

**What's Backed Up:**
- ✅ All Docker container configurations
- ✅ Nginx Proxy Manager configs & SSL certificates
- ✅ Headscale database & config
- ✅ All dashboard settings (Heimdall, Organizr, Uptime Kuma)
- ✅ All service configs
- ❌ Ollama models (re-downloadable)
- ❌ Cache files
- ❌ Log files

## Automated Backups

**Schedule:** Daily at 3:00 AM (configured in crontab)

**View Backup Logs:**
```bash
# Real-time logs
docker logs -f maintenance

# Backup script logs
cat ~/docker-data/maintenance/logs/backup-configs.log
```

**List Existing Backups:**
```bash
ls -lh /mnt/media/backups/docker-configs/
```

## Manual Backup

Run a backup anytime:
```bash
docker exec maintenance /scripts/backup-configs.sh
```

## Restore from Backup

### Full Restore

1. **Stop all containers:**
   ```bash
   docker stop $(docker ps -aq)
   ```

2. **Backup current state (just in case):**
   ```bash
   mv ~/docker-data ~/docker-data.old
   ```

3. **Extract backup:**
   ```bash
   cd ~
   tar -xzf /mnt/media/backups/docker-configs/docker-configs-YYYYMMDD-HHMMSS.tar.gz
   ```

4. **Restart containers:**
   ```bash
   docker start $(docker ps -aq)
   ```

5. **Verify services:**
   ```bash
   docker ps
   ```

### Selective Restore (Single Service)

Restore only one service's config (example: Headscale):

```bash
# Extract only headscale directory
tar -xzf /mnt/media/backups/docker-configs/docker-configs-20251111-221349.tar.gz \
    --strip-components=2 \
    -C ~/docker-data/ \
    docker-data/headscale

# Restart the service
docker restart headscale
```

## Adding New Maintenance Tasks

The maintenance container can run any scheduled task, not just backups.

### 1. Create New Script

```bash
# Create script file
nano ~/docker-data/maintenance/scripts/my-task.sh

# Make it executable
chmod +x ~/docker-data/maintenance/scripts/my-task.sh
```

### 2. Add to Crontab

```bash
# Edit crontab
nano ~/docker-data/maintenance/crontab

# Add your schedule (example: every Sunday at 4 AM)
# 0 4 * * 0 /scripts/my-task.sh
```

### 3. Restart Container

```bash
docker restart maintenance
```

### Examples of Future Tasks

- **Weekly cleanup:** Remove old Docker images
- **Health checks:** Verify all services are responding
- **Update checks:** Notify when container updates available
- **Database optimization:** Compact/optimize databases
- **SSL renewal checks:** Verify certificates are valid

## Testing Backup Integrity

Periodically test that backups can be restored:

```bash
# Create test directory
mkdir -p /tmp/backup-test

# Extract backup
tar -xzf /mnt/media/backups/docker-configs/docker-configs-LATEST.tar.gz \
    -C /tmp/backup-test

# Verify contents
ls -la /tmp/backup-test/docker-data/

# Clean up
rm -rf /tmp/backup-test
```

## Troubleshooting

### Backup Not Running

**Check if container is running:**
```bash
docker ps | grep maintenance
```

**Check cron logs:**
```bash
docker logs maintenance
```

**Manually run backup to test:**
```bash
docker exec maintenance /scripts/backup-configs.sh
```

### Backup Taking Too Long

- Check if exclusions are working (Ollama models should be excluded)
- Monitor disk I/O: `iostat -x 1`
- Check HDD health: `sudo smartctl -a /dev/sdb`

### Backup Disk Full

- Old backups auto-delete after 30 days
- Manually remove old backups if needed:
  ```bash
  # List backups by size
  du -h /mnt/media/backups/docker-configs/*

  # Remove specific backup
  rm /mnt/media/backups/docker-configs/docker-configs-20251001-*.tar.gz
  ```

### Restore Failed

1. Check backup file integrity:
   ```bash
   tar -tzf /mnt/media/backups/docker-configs/backup-file.tar.gz > /dev/null
   ```

2. If corrupted, try previous backup

3. Check disk space before restoring:
   ```bash
   df -h ~/docker-data
   ```

## Backup Storage

**Current Usage:**
- ~94 MB per daily backup
- 30 days retention = ~2.8 GB total
- Stored on 3.6 TB HDD (plenty of space)

**Offsite Backups (Recommended):**

For extra protection, periodically copy backups to external drive:

```bash
# Copy last 7 days to external drive
rsync -av --progress /mnt/media/backups/docker-configs/ /mnt/external-drive/backups/
```

---

**Last Updated:** 2025-11-11