From cfb6cea452482fa43022cb90aeb495c8770b07a6 Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Thu, 26 Feb 2026 10:35:09 +0100 Subject: [PATCH] docs: add project handover notes Co-Authored-By: Claude Opus 4.6 --- HANDOVER-swap-limits.md | 119 +++++++++++++++++++++++++++++ PROJECT_CLAUDIFICATION_HANDOVER.md | 102 +++++++++++++++++++++++++ 2 files changed, 221 insertions(+) create mode 100644 HANDOVER-swap-limits.md create mode 100644 PROJECT_CLAUDIFICATION_HANDOVER.md diff --git a/HANDOVER-swap-limits.md b/HANDOVER-swap-limits.md new file mode 100644 index 0000000..1dbfcd1 --- /dev/null +++ b/HANDOVER-swap-limits.md @@ -0,0 +1,119 @@ +# Handover: Add Swap Limits to Containers + +## Problem + +System swap is at 99% (2043/2047 MB) after 60 days uptime. All containers have `mem_swappiness` and `memswap_limit` unset, meaning when they hit their memory ceiling the kernel pushes pages to swap indefinitely instead of applying back-pressure. Over time this exhausts swap. + +## Root Cause Analysis (2026-02-03) + +Per-process swap audit identified these containers as the top offenders: + +| Container | Swap Used | Mem Limit | File | +|-----------|-----------|-----------|------| +| authentik-worker | ~865 MB (across 5 worker procs) | 1G | `stacks/authentik.yml` | +| neo4j | 384 MB | 4G | `stacks/neo4j.yml` | +| penpot-backend | 372 MB | 1G | `stacks/penpot.yml` | +| authentik-server | 248 MB (gunicorn master+worker) | 512M | `stacks/authentik.yml` | +| open-webui | 129 MB | 1G | `stacks/open-webui.yml` | +| library-desk | 130 MB | 768M | `stacks/library-desk.yml` | + +Additionally, host-level `clamd` (ClamAV) uses ~410 MB swap but is not managed by Portainer. + +## Required Changes + +For each container listed above, add `memswap_limit` equal to the `memory` limit and set `mem_swappiness: 0` under the top-level service config (not under `deploy`). This is Docker Compose v2 compatible syntax that works alongside v3 deploy blocks. + +### stacks/authentik.yml + +**authentik-server** — add to the service block (sibling to `deploy`, not nested inside it): + +```yaml + server: + # ... existing config ... + mem_swappiness: 0 + memswap_limit: 512M +``` + +**authentik-worker** — same pattern: + +```yaml + worker: + # ... existing config ... + mem_swappiness: 0 + memswap_limit: 1G +``` + +### stacks/neo4j.yml + +```yaml + neo4j: + # ... existing config ... + mem_swappiness: 0 + memswap_limit: 4G +``` + +### stacks/penpot.yml + +**penpot-backend** only (frontend and exporter are not swap offenders): + +```yaml + penpot-backend: + # ... existing config ... + mem_swappiness: 0 + memswap_limit: 1G +``` + +### stacks/open-webui.yml + +```yaml + open-webui: + # ... existing config ... + mem_swappiness: 0 + memswap_limit: 1G +``` + +### stacks/library-desk.yml + +```yaml + library-desk: + # ... existing config ... + mem_swappiness: 0 + memswap_limit: 768M +``` + +## What These Settings Do + +- `memswap_limit: X` — total memory+swap budget. Setting it equal to the memory limit means zero swap allowed. +- `mem_swappiness: 0` — tells the kernel to avoid swapping for this container unless absolutely necessary. + +Together, these ensure containers are constrained to their RAM allocation. If they exceed it, the OOM killer handles it properly rather than silently filling swap. + +## Deployment Notes + +- These are runtime container settings — each stack needs to be redeployed in Portainer (or via `docker compose up -d`) for changes to take effect. +- After deploying, clear current swap: `sudo swapoff -a && sudo swapon -a` +- Monitor with: `cd /mnt/media/Projects/system-management && ./sysmon check memory` +- If any container starts OOM-killing after this change, its memory limit may need bumping. The most likely candidate is `authentik-worker` (was using 865 MB swap on top of its 1G RAM limit — may need 1.5G or 2G). + +## ClamAV (Host Service) + +ClamAV (`clamd`) is the single largest swap consumer at 410 MB. It runs as a host service, not a container. To limit it, edit `/etc/clamav/clamd.conf` or the systemd unit to set a memory ceiling. This is outside the scope of portainer-core but noted for completeness. + +## Verification + +After all stacks are redeployed and swap is cleared: + +```bash +# Confirm swap is back to healthy levels +free -h + +# Run system check +cd /mnt/media/Projects/system-management && ./sysmon check memory + +# Verify no container is using swap +for cid in $(docker ps -q); do + name=$(docker inspect --format '{{.Name}}' "$cid") + swap=$(docker stats --no-stream --format '{{.MemUsage}}' "$cid") + echo "$name: $swap" +done +``` diff --git a/PROJECT_CLAUDIFICATION_HANDOVER.md b/PROJECT_CLAUDIFICATION_HANDOVER.md new file mode 100644 index 0000000..c2b1606 --- /dev/null +++ b/PROJECT_CLAUDIFICATION_HANDOVER.md @@ -0,0 +1,102 @@ +# Tatlock Claudification Handover - portainer-core + +## Context + +Tatlock (the butler) is being upgraded to use Claude as its primary LLM backend instead of Ollama. This requires infrastructure changes to support the new configuration. + +**Parent Issue:** See `/mnt/media/Projects/tatlock/PROJECT_CLAUDIFICATION.md` + +## Impact on portainer-core + +Portainer-core manages Docker Swarm stacks. The Tatlock stack needs updated configuration. + +## Required Changes + +### Priority: High (Required for Claude backend) + +- [ ] **Update `stacks/agents.yml`** (or wherever Tatlock is defined) + + Add new environment variables: + ```yaml + services: + tatlock: + environment: + # Anthropic Configuration (Claude - preferred backend) + - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} + - ANTHROPIC_MODEL=claude-sonnet-4-20250514 + - PREFER_CLOUD_BACKEND=true + # Existing Ollama config remains as fallback + - OLLAMA_HOST=http://ollama:11434 + - OLLAMA_DEFAULT_MODEL=mistral-nemo:latest + ``` + +- [ ] **Configure secrets management** + + Options: + 1. **Docker secrets** (recommended for Swarm): + ```yaml + secrets: + anthropic_api_key: + external: true + services: + tatlock: + secrets: + - anthropic_api_key + environment: + - ANTHROPIC_API_KEY_FILE=/run/secrets/anthropic_api_key + ``` + + 2. **Environment file** (simpler but less secure): + ```yaml + services: + tatlock: + env_file: + - ./secrets/tatlock.env + ``` + +- [ ] **Update `CONTAINERS.md`** + - Document new environment variables + - Note Claude as preferred backend with Ollama fallback + - Update any API documentation + +### Priority: Medium (Phase 2 - MCP Server) + +- [ ] **Add `tatlock-mcp` service definition** (when Phase 2 is ready) + ```yaml + tatlock-mcp: + image: git.schweitz.net/jpmschweitzer/tatlock:latest + command: ["python", "-m", "src.mcp.server"] + ports: + - "8778:8778" + environment: + - MCP_AUTH_TOKEN=${MCP_AUTH_TOKEN} + # ... other config + ``` + +### Priority: Low (Optimization) + +- [ ] **Review resource limits** + - Claude backend may have different resource profile than Ollama + - Monitor memory/CPU after deployment + - Adjust limits if needed + +## Testing + +After stack update: + +```bash +# Check Tatlock logs for backend selection +docker service logs tatlock_tatlock 2>&1 | grep -E "claude|backend" + +# Should see: +# model_backend_configured backend=claude model=claude-sonnet-4-20250514 +# OR (if no API key): +# claude_health_check_skipped reason=no_api_key +# model_backend_configured backend=ollama +``` + +## Timeline + +- **Blocking:** Yes - required for production Claude deployment +- **When to implement:** When ANTHROPIC_API_KEY is available +- **Effort:** ~1 hour for stack updates, ~30 min for secrets setup