# Handover: Add Swap Limits to Containers

## Problem

System swap is at 99% (2043/2047 MB) after 60 days uptime. All containers have `mem_swappiness` and `memswap_limit` unset, meaning when they hit their memory ceiling the kernel pushes pages to swap indefinitely instead of applying back-pressure. Over time this exhausts swap.

## Root Cause Analysis (2026-02-03)

Per-process swap audit identified these containers as the top offenders:

| Container | Swap Used | Mem Limit | File |
|-----------|-----------|-----------|------|
| authentik-worker | ~865 MB (across 5 worker procs) | 1G | `stacks/authentik.yml` |
| neo4j | 384 MB | 4G | `stacks/neo4j.yml` |
| penpot-backend | 372 MB | 1G | `stacks/penpot.yml` |
| authentik-server | 248 MB (gunicorn master+worker) | 512M | `stacks/authentik.yml` |
| open-webui | 129 MB | 1G | `stacks/open-webui.yml` |
| library-desk | 130 MB | 768M | `stacks/library-desk.yml` |

Additionally, host-level `clamd` (ClamAV) uses ~410 MB swap but is not managed by Portainer.

## Required Changes

For each container listed above, add `memswap_limit` equal to the `memory` limit and set `mem_swappiness: 0` under the top-level service config (not under `deploy`). This is Docker Compose v2 compatible syntax that works alongside v3 deploy blocks.

### stacks/authentik.yml

**authentik-server** — add to the service block (sibling to `deploy`, not nested inside it):

```yaml
  server:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 512M
```

**authentik-worker** — same pattern:

```yaml
  worker:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 1G
```

### stacks/neo4j.yml

```yaml
  neo4j:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 4G
```

### stacks/penpot.yml

**penpot-backend** only (frontend and exporter are not swap offenders):

```yaml
  penpot-backend:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 1G
```

### stacks/open-webui.yml

```yaml
  open-webui:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 1G
```

### stacks/library-desk.yml

```yaml
  library-desk:
    # ... existing config ...
    mem_swappiness: 0
    memswap_limit: 768M
```

## What These Settings Do

- `memswap_limit: X` — total memory+swap budget. Setting it equal to the memory limit means zero swap allowed.
- `mem_swappiness: 0` — tells the kernel to avoid swapping for this container unless absolutely necessary.

Together, these ensure containers are constrained to their RAM allocation. If they exceed it, the OOM killer handles it properly rather than silently filling swap.

## Deployment Notes

- These are runtime container settings — each stack needs to be redeployed in Portainer (or via `docker compose up -d`) for changes to take effect.
- After deploying, clear current swap: `sudo swapoff -a && sudo swapon -a`
- Monitor with: `cd /mnt/media/Projects/system-management && ./sysmon check memory`
- If any container starts OOM-killing after this change, its memory limit may need bumping. The most likely candidate is `authentik-worker` (was using 865 MB swap on top of its 1G RAM limit — may need 1.5G or 2G).

## ClamAV (Host Service)

ClamAV (`clamd`) is the single largest swap consumer at 410 MB. It runs as a host service, not a container. To limit it, edit `/etc/clamav/clamd.conf` or the systemd unit to set a memory ceiling. This is outside the scope of portainer-core but noted for completeness.

## Verification

After all stacks are redeployed and swap is cleared:

```bash
# Confirm swap is back to healthy levels
free -h

# Run system check
cd /mnt/media/Projects/system-management && ./sysmon check memory

# Verify no container is using swap
for cid in $(docker ps -q); do
  name=$(docker inspect --format '{{.Name}}' "$cid")
  swap=$(docker stats --no-stream --format '{{.MemUsage}}' "$cid")
  echo "$name: $swap"
done
```