portainer-core/HANDOVER-swap-limits.md

Handover: Add Swap Limits to Containers

Problem

System swap is at 99% (2043/2047 MB) after 60 days uptime. All containers have mem_swappiness and memswap_limit unset, meaning when they hit their memory ceiling the kernel pushes pages to swap indefinitely instead of applying back-pressure. Over time this exhausts swap.

Root Cause Analysis (2026-02-03)

Per-process swap audit identified these containers as the top offenders:

Container	Swap Used	Mem Limit	File
authentik-worker	~865 MB (across 5 worker procs)	1G	stacks/authentik.yml
neo4j	384 MB	4G	stacks/neo4j.yml
penpot-backend	372 MB	1G	stacks/penpot.yml
authentik-server	248 MB (gunicorn master+worker)	512M	stacks/authentik.yml
open-webui	129 MB	1G	stacks/open-webui.yml
library-desk	130 MB	768M	stacks/library-desk.yml

Additionally, host-level clamd (ClamAV) uses ~410 MB swap but is not managed by Portainer.

Required Changes

For each container listed above, add memswap_limit equal to the memory limit and set mem_swappiness: 0 under the top-level service config (not under deploy). This is Docker Compose v2 compatible syntax that works alongside v3 deploy blocks.

stacks/authentik.yml

authentik-server — add to the service block (sibling to deploy, not nested inside it):

  server:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 512M

authentik-worker — same pattern:

  worker:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 1G

stacks/neo4j.yml

  neo4j:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 4G

stacks/penpot.yml

penpot-backend only (frontend and exporter are not swap offenders):

  penpot-backend:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 1G

stacks/open-webui.yml

  open-webui:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 1G

stacks/library-desk.yml

  library-desk:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 768M

What These Settings Do

• memswap_limit: X — total memory+swap budget. Setting it equal to the memory limit means zero swap allowed.
• mem_swappiness: 0 — tells the kernel to avoid swapping for this container unless absolutely necessary.

Together, these ensure containers are constrained to their RAM allocation. If they exceed it, the OOM killer handles it properly rather than silently filling swap.

Deployment Notes

• These are runtime container settings — each stack needs to be redeployed in Portainer (or via docker compose up -d) for changes to take effect.
• After deploying, clear current swap: sudo swapoff -a && sudo swapon -a
• Monitor with: cd /mnt/media/Projects/system-management && ./sysmon check memory
• If any container starts OOM-killing after this change, its memory limit may need bumping. The most likely candidate is authentik-worker (was using 865 MB swap on top of its 1G RAM limit — may need 1.5G or 2G).

ClamAV (Host Service)

ClamAV (clamd) is the single largest swap consumer at 410 MB. It runs as a host service, not a container. To limit it, edit /etc/clamav/clamd.conf or the systemd unit to set a memory ceiling. This is outside the scope of portainer-core but noted for completeness.

Verification

After all stacks are redeployed and swap is cleared:

# Confirm swap is back to healthy levels
free -h

# Run system check
cd /mnt/media/Projects/system-management && ./sysmon check memory

# Verify no container is using swap
for cid in $(docker ps -q); do
  name=$(docker inspect --format '{{.Name}}' "$cid")
  swap=$(docker stats --no-stream --format '{{.MemUsage}}' "$cid")
  echo "$name: $swap"
done