portainer-core/docs/npm-logging-guide.md

# NPM Logging & Audit Guide

> Centralized logging for all external access
> Created: 2025-11-11

## Why NPM is the Logging Hub

**All public service access routes through NPM**, which means:
- Every external request is logged
- Failed authentication attempts tracked
- Rate limiting violations recorded
- SSL certificate renewals logged
- Configuration changes audited

## Accessing NPM Logs

### Via NPM UI

**Access NPM admin panel:**
- VPN: http://10.99.0.1:81
- Local: http://192.168.86.149:81

**View logs:**
1. Navigate to each Proxy Host
2. Click "View Logs" button
3. See real-time access logs
4. Filter by status code, IP, user agent

### Via Docker Logs

**Real-time monitoring:**
```bash
# All NPM logs
docker logs -f nginx-proxy-manager

# Filter for access logs only
docker logs -f nginx-proxy-manager 2>&1 | grep -i "access"

# Filter for errors
docker logs -f nginx-proxy-manager 2>&1 | grep -i "error"

# Filter for specific service (e.g., Jellyfin)
docker logs -f nginx-proxy-manager 2>&1 | grep "media.schweitz.net"
```

### Via Log Files

**Log location:**
```bash
# Access logs stored in container volume
ls -lh ~/docker-data/nginx-proxy-manager/data/logs/

# View access logs
tail -f ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log

# View error logs
tail -f ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*_error.log
```

## Log Format

**Standard NPM access log:**
```
192.0.2.1 - - [11/Nov/2025:21:30:15 +0100] "GET /api/users HTTP/2.0" 200 1234 "https://media.schweitz.net" "Mozilla/5.0..."
```

**Fields:**
- **IP address**: Client IP (or Cloudflare IP if proxied)
- **Timestamp**: When request occurred
- **HTTP method**: GET, POST, etc.
- **Request path**: /api/users
- **Protocol**: HTTP/2.0
- **Status code**: 200 (success), 404 (not found), 403 (forbidden), etc.
- **Bytes sent**: Response size
- **Referer**: Previous page
- **User agent**: Browser/client info

## Useful Log Queries

### Find Failed Login Attempts
```bash
# Status codes 401 (unauthorized) or 403 (forbidden)
grep -E " (401|403) " ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log

# With IP addresses
grep -E " (401|403) " ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log | awk '{print $1}' | sort | uniq -c | sort -nr
```

### Monitor Specific Service Access
```bash
# Jellyfin access
grep "media.schweitz.net" ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log | tail -20

# Nextcloud uploads (POST requests)
grep "cloud.schweitz.net" ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log | grep "POST"
```

### Identify High-Traffic IPs
```bash
# Top 10 IP addresses by request count
awk '{print $1}' ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log | sort | uniq -c | sort -nr | head -10
```

### Monitor SSL Certificate Activity
```bash
# Certificate renewal attempts
docker logs nginx-proxy-manager 2>&1 | grep -i "letsencrypt"

# Certificate errors
docker logs nginx-proxy-manager 2>&1 | grep -i "certificate" | grep -i "error"
```

### Track API Usage
```bash
# API endpoint access
grep "/api/" ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log

# Specific API endpoint
grep "/api/login" ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log
```

## Security Monitoring

### Suspicious Activity Patterns

**Brute force attempts:**
```bash
# Multiple 401s from same IP (potential brute force)
grep " 401 " ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log | \
  awk '{print $1}' | sort | uniq -c | sort -nr | \
  awk '$1 > 10 {print "Potential brute force from " $2 " (" $1 " attempts)"}'
```

**Directory scanning:**
```bash
# Looking for 404s (scanning for vulnerabilities)
grep " 404 " ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log | \
  grep -E "(wp-admin|phpmyadmin|admin|login\.php)"
```

**Unusual user agents:**
```bash
# Non-browser requests (potential bots/scrapers)
grep -v "Mozilla" ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log | \
  grep -v "curl" | tail -20
```

## Log Rotation

**Automatic rotation configuration:**

NPM handles basic rotation, but for long-term storage:

```bash
# Create logrotate config
sudo tee /etc/logrotate.d/nginx-proxy-manager <<EOF
/home/jpmschweitzer/docker-data/nginx-proxy-manager/data/logs/*.log {
    daily
    rotate 30
    compress
    delaycompress
    notifempty
    missingok
    create 0644 root root
    postrotate
        docker exec nginx-proxy-manager nginx -s reload > /dev/null 2>&1 || true
    endscript
}
EOF

# Test logrotate config
sudo logrotate -d /etc/logrotate.d/nginx-proxy-manager
```

**Manual log cleanup:**
```bash
# Archive old logs
cd ~/docker-data/nginx-proxy-manager/data/logs/
tar -czf logs-archive-$(date +%Y%m%d).tar.gz *.log
mv logs-archive-*.tar.gz ~/backups/

# Clean logs older than 30 days
find ~/docker-data/nginx-proxy-manager/data/logs/ -name "*.log" -mtime +30 -delete
```

## Centralized Logging (Future Enhancement)

**Option 1: Ship logs to external service**

Use a log aggregator like:
- Loki + Grafana (self-hosted)
- Elasticsearch + Kibana
- Splunk
- Cloud services (Datadog, Loggly)

**Option 2: Syslog forwarding**

Configure NPM to forward to syslog:
```nginx
# Add to NPM custom nginx config
access_log syslog:server=10.99.0.1:514,tag=nginx combined;
```

**Option 3: Promtail + Loki (Recommended)**

Deploy Promtail container to tail NPM logs and send to Loki:
```yaml
# Future: stacks/promtail.yml
services:
  promtail:
    image: grafana/promtail:latest
    volumes:
      - /home/jpmschweitzer/docker-data/nginx-proxy-manager/data/logs:/var/log/nginx
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
```

## Compliance Logging

**For audit trails, log these events:**

### Access Events
- ✅ All successful logins (200 responses to /login endpoints)
- ✅ Failed login attempts (401/403 responses)
- ✅ File downloads (GET requests with large response sizes)
- ✅ File uploads (POST/PUT requests)
- ✅ API calls (requests to /api/ paths)

### Security Events
- ✅ SSL certificate renewals
- ✅ Configuration changes in NPM
- ✅ Rate limit violations
- ✅ Blocked IPs (403 responses)

### Monitoring Events
- ✅ Service downtime (502/503 responses)
- ✅ Slow responses (response time tracking)
- ✅ High traffic patterns

## Alerting Setup

**Create alerts for critical events:**

### Using Uptime Kuma

Configure HTTP(s) monitors in Uptime Kuma:
- Monitor each public service
- Alert on downtime
- Track response times

### Using Custom Scripts

**Example: Alert on failed logins:**
```bash
#!/bin/bash
# /home/jpmschweitzer/scripts/monitor-failed-logins.sh

THRESHOLD=10
LOG_FILE="/home/jpmschweitzer/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log"

# Count 401s in last 5 minutes
RECENT_FAILURES=$(find ~/docker-data/nginx-proxy-manager/data/logs/ -name "proxy-host-*.log" -mmin -5 -exec grep -c " 401 " {} + | awk '{s+=$1} END {print s}')

if [ "$RECENT_FAILURES" -gt "$THRESHOLD" ]; then
    echo "ALERT: $RECENT_FAILURES failed login attempts in last 5 minutes" | \
        mail -s "Security Alert: High Failed Login Rate" admin@schweitz.net
fi
```

**Run via cron:**
```cron
*/5 * * * * /home/jpmschweitzer/scripts/monitor-failed-logins.sh
```

## Performance Monitoring

**Track service performance via logs:**

### Response Time Analysis
```bash
# Extract response times (if configured in NPM)
grep "upstream_response_time" ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log | \
  awk '{print $NF}' | sort -n | tail -20
```

### Bandwidth Usage
```bash
# Sum bytes sent per service
awk '{sum+=$10} END {print "Total bytes: " sum " (" sum/1024/1024 " MB)"}' \
  ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-media*.log
```

### Most Accessed Endpoints
```bash
# Top 10 requested paths
awk '{print $7}' ~/docker-data/nginx-proxy-manager/data/logs/proxy-host-*.log | \
  sort | uniq -c | sort -nr | head -10
```

## Best Practices

### Regular Log Review
- ✅ Check NPM logs weekly for suspicious activity
- ✅ Review SSL certificate status monthly
- ✅ Archive logs older than 30 days
- ✅ Monitor for unusual traffic patterns

### Retention Policy
- **Active logs**: 30 days (on SSD)
- **Compressed archives**: 1 year (on HDD /mnt/media/backups/logs/)
- **Long-term storage**: Ship to external service if needed

### Privacy Considerations
- ⚠️ Logs contain IP addresses (PII in EU)
- ⚠️ Don't log full request bodies (may contain passwords)
- ⚠️ Rotate/delete old logs per privacy policy
- ⚠️ Secure log access (only admins via VPN)

---

**Summary:**
- All external access logged in NPM
- Logs accessible via UI, Docker, or files
- Use for security monitoring and audit trails
- Set up alerts for critical events
- Regular log review and rotation