120 lines
3.8 KiB
Markdown
120 lines
3.8 KiB
Markdown
# Handover: Add Swap Limits to Containers
|
|
|
|
## Problem
|
|
|
|
System swap is at 99% (2043/2047 MB) after 60 days uptime. All containers have `mem_swappiness` and `memswap_limit` unset, meaning when they hit their memory ceiling the kernel pushes pages to swap indefinitely instead of applying back-pressure. Over time this exhausts swap.
|
|
|
|
## Root Cause Analysis (2026-02-03)
|
|
|
|
Per-process swap audit identified these containers as the top offenders:
|
|
|
|
| Container | Swap Used | Mem Limit | File |
|
|
|-----------|-----------|-----------|------|
|
|
| authentik-worker | ~865 MB (across 5 worker procs) | 1G | `stacks/authentik.yml` |
|
|
| neo4j | 384 MB | 4G | `stacks/neo4j.yml` |
|
|
| penpot-backend | 372 MB | 1G | `stacks/penpot.yml` |
|
|
| authentik-server | 248 MB (gunicorn master+worker) | 512M | `stacks/authentik.yml` |
|
|
| open-webui | 129 MB | 1G | `stacks/open-webui.yml` |
|
|
| library-desk | 130 MB | 768M | `stacks/library-desk.yml` |
|
|
|
|
Additionally, host-level `clamd` (ClamAV) uses ~410 MB swap but is not managed by Portainer.
|
|
|
|
## Required Changes
|
|
|
|
For each container listed above, add `memswap_limit` equal to the `memory` limit and set `mem_swappiness: 0` under the top-level service config (not under `deploy`). This is Docker Compose v2 compatible syntax that works alongside v3 deploy blocks.
|
|
|
|
### stacks/authentik.yml
|
|
|
|
**authentik-server** — add to the service block (sibling to `deploy`, not nested inside it):
|
|
|
|
```yaml
|
|
server:
|
|
# ... existing config ...
|
|
mem_swappiness: 0
|
|
memswap_limit: 512M
|
|
```
|
|
|
|
**authentik-worker** — same pattern:
|
|
|
|
```yaml
|
|
worker:
|
|
# ... existing config ...
|
|
mem_swappiness: 0
|
|
memswap_limit: 1G
|
|
```
|
|
|
|
### stacks/neo4j.yml
|
|
|
|
```yaml
|
|
neo4j:
|
|
# ... existing config ...
|
|
mem_swappiness: 0
|
|
memswap_limit: 4G
|
|
```
|
|
|
|
### stacks/penpot.yml
|
|
|
|
**penpot-backend** only (frontend and exporter are not swap offenders):
|
|
|
|
```yaml
|
|
penpot-backend:
|
|
# ... existing config ...
|
|
mem_swappiness: 0
|
|
memswap_limit: 1G
|
|
```
|
|
|
|
### stacks/open-webui.yml
|
|
|
|
```yaml
|
|
open-webui:
|
|
# ... existing config ...
|
|
mem_swappiness: 0
|
|
memswap_limit: 1G
|
|
```
|
|
|
|
### stacks/library-desk.yml
|
|
|
|
```yaml
|
|
library-desk:
|
|
# ... existing config ...
|
|
mem_swappiness: 0
|
|
memswap_limit: 768M
|
|
```
|
|
|
|
## What These Settings Do
|
|
|
|
- `memswap_limit: X` — total memory+swap budget. Setting it equal to the memory limit means zero swap allowed.
|
|
- `mem_swappiness: 0` — tells the kernel to avoid swapping for this container unless absolutely necessary.
|
|
|
|
Together, these ensure containers are constrained to their RAM allocation. If they exceed it, the OOM killer handles it properly rather than silently filling swap.
|
|
|
|
## Deployment Notes
|
|
|
|
- These are runtime container settings — each stack needs to be redeployed in Portainer (or via `docker compose up -d`) for changes to take effect.
|
|
- After deploying, clear current swap: `sudo swapoff -a && sudo swapon -a`
|
|
- Monitor with: `cd /mnt/media/Projects/system-management && ./sysmon check memory`
|
|
- If any container starts OOM-killing after this change, its memory limit may need bumping. The most likely candidate is `authentik-worker` (was using 865 MB swap on top of its 1G RAM limit — may need 1.5G or 2G).
|
|
|
|
## ClamAV (Host Service)
|
|
|
|
ClamAV (`clamd`) is the single largest swap consumer at 410 MB. It runs as a host service, not a container. To limit it, edit `/etc/clamav/clamd.conf` or the systemd unit to set a memory ceiling. This is outside the scope of portainer-core but noted for completeness.
|
|
|
|
## Verification
|
|
|
|
After all stacks are redeployed and swap is cleared:
|
|
|
|
```bash
|
|
# Confirm swap is back to healthy levels
|
|
free -h
|
|
|
|
# Run system check
|
|
cd /mnt/media/Projects/system-management && ./sysmon check memory
|
|
|
|
# Verify no container is using swap
|
|
for cid in $(docker ps -q); do
|
|
name=$(docker inspect --format '{{.Name}}' "$cid")
|
|
swap=$(docker stats --no-stream --format '{{.MemUsage}}' "$cid")
|
|
echo "$name: $swap"
|
|
done
|
|
```
|