docs: add project handover notes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.6
@@ -0,0 +1,119 @@
|
||||
# Handover: Add Swap Limits to Containers
|
||||
|
||||
## Problem
|
||||
|
||||
System swap is at 99% (2043/2047 MB) after 60 days uptime. All containers have `mem_swappiness` and `memswap_limit` unset, meaning when they hit their memory ceiling the kernel pushes pages to swap indefinitely instead of applying back-pressure. Over time this exhausts swap.
|
||||
|
||||
## Root Cause Analysis (2026-02-03)
|
||||
|
||||
Per-process swap audit identified these containers as the top offenders:
|
||||
|
||||
| Container | Swap Used | Mem Limit | File |
|
||||
|-----------|-----------|-----------|------|
|
||||
| authentik-worker | ~865 MB (across 5 worker procs) | 1G | `stacks/authentik.yml` |
|
||||
| neo4j | 384 MB | 4G | `stacks/neo4j.yml` |
|
||||
| penpot-backend | 372 MB | 1G | `stacks/penpot.yml` |
|
||||
| authentik-server | 248 MB (gunicorn master+worker) | 512M | `stacks/authentik.yml` |
|
||||
| open-webui | 129 MB | 1G | `stacks/open-webui.yml` |
|
||||
| library-desk | 130 MB | 768M | `stacks/library-desk.yml` |
|
||||
|
||||
Additionally, host-level `clamd` (ClamAV) uses ~410 MB swap but is not managed by Portainer.
|
||||
|
||||
## Required Changes
|
||||
|
||||
For each container listed above, add `memswap_limit` equal to the `memory` limit and set `mem_swappiness: 0` under the top-level service config (not under `deploy`). This is Docker Compose v2 compatible syntax that works alongside v3 deploy blocks.
|
||||
|
||||
### stacks/authentik.yml
|
||||
|
||||
**authentik-server** — add to the service block (sibling to `deploy`, not nested inside it):
|
||||
|
||||
```yaml
|
||||
server:
|
||||
# ... existing config ...
|
||||
mem_swappiness: 0
|
||||
memswap_limit: 512M
|
||||
```
|
||||
|
||||
**authentik-worker** — same pattern:
|
||||
|
||||
```yaml
|
||||
worker:
|
||||
# ... existing config ...
|
||||
mem_swappiness: 0
|
||||
memswap_limit: 1G
|
||||
```
|
||||
|
||||
### stacks/neo4j.yml
|
||||
|
||||
```yaml
|
||||
neo4j:
|
||||
# ... existing config ...
|
||||
mem_swappiness: 0
|
||||
memswap_limit: 4G
|
||||
```
|
||||
|
||||
### stacks/penpot.yml
|
||||
|
||||
**penpot-backend** only (frontend and exporter are not swap offenders):
|
||||
|
||||
```yaml
|
||||
penpot-backend:
|
||||
# ... existing config ...
|
||||
mem_swappiness: 0
|
||||
memswap_limit: 1G
|
||||
```
|
||||
|
||||
### stacks/open-webui.yml
|
||||
|
||||
```yaml
|
||||
open-webui:
|
||||
# ... existing config ...
|
||||
mem_swappiness: 0
|
||||
memswap_limit: 1G
|
||||
```
|
||||
|
||||
### stacks/library-desk.yml
|
||||
|
||||
```yaml
|
||||
library-desk:
|
||||
# ... existing config ...
|
||||
mem_swappiness: 0
|
||||
memswap_limit: 768M
|
||||
```
|
||||
|
||||
## What These Settings Do
|
||||
|
||||
- `memswap_limit: X` — total memory+swap budget. Setting it equal to the memory limit means zero swap allowed.
|
||||
- `mem_swappiness: 0` — tells the kernel to avoid swapping for this container unless absolutely necessary.
|
||||
|
||||
Together, these ensure containers are constrained to their RAM allocation. If they exceed it, the OOM killer handles it properly rather than silently filling swap.
|
||||
|
||||
## Deployment Notes
|
||||
|
||||
- These are runtime container settings — each stack needs to be redeployed in Portainer (or via `docker compose up -d`) for changes to take effect.
|
||||
- After deploying, clear current swap: `sudo swapoff -a && sudo swapon -a`
|
||||
- Monitor with: `cd /mnt/media/Projects/system-management && ./sysmon check memory`
|
||||
- If any container starts OOM-killing after this change, its memory limit may need bumping. The most likely candidate is `authentik-worker` (was using 865 MB swap on top of its 1G RAM limit — may need 1.5G or 2G).
|
||||
|
||||
## ClamAV (Host Service)
|
||||
|
||||
ClamAV (`clamd`) is the single largest swap consumer at 410 MB. It runs as a host service, not a container. To limit it, edit `/etc/clamav/clamd.conf` or the systemd unit to set a memory ceiling. This is outside the scope of portainer-core but noted for completeness.
|
||||
|
||||
## Verification
|
||||
|
||||
After all stacks are redeployed and swap is cleared:
|
||||
|
||||
```bash
|
||||
# Confirm swap is back to healthy levels
|
||||
free -h
|
||||
|
||||
# Run system check
|
||||
cd /mnt/media/Projects/system-management && ./sysmon check memory
|
||||
|
||||
# Verify no container is using swap
|
||||
for cid in $(docker ps -q); do
|
||||
name=$(docker inspect --format '{{.Name}}' "$cid")
|
||||
swap=$(docker stats --no-stream --format '{{.MemUsage}}' "$cid")
|
||||
echo "$name: $swap"
|
||||
done
|
||||
```
|
||||
@@ -0,0 +1,102 @@
|
||||
# Tatlock Claudification Handover - portainer-core
|
||||
|
||||
## Context
|
||||
|
||||
Tatlock (the butler) is being upgraded to use Claude as its primary LLM backend instead of Ollama. This requires infrastructure changes to support the new configuration.
|
||||
|
||||
**Parent Issue:** See `/mnt/media/Projects/tatlock/PROJECT_CLAUDIFICATION.md`
|
||||
|
||||
## Impact on portainer-core
|
||||
|
||||
Portainer-core manages Docker Swarm stacks. The Tatlock stack needs updated configuration.
|
||||
|
||||
## Required Changes
|
||||
|
||||
### Priority: High (Required for Claude backend)
|
||||
|
||||
- [ ] **Update `stacks/agents.yml`** (or wherever Tatlock is defined)
|
||||
|
||||
Add new environment variables:
|
||||
```yaml
|
||||
services:
|
||||
tatlock:
|
||||
environment:
|
||||
# Anthropic Configuration (Claude - preferred backend)
|
||||
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
|
||||
- ANTHROPIC_MODEL=claude-sonnet-4-20250514
|
||||
- PREFER_CLOUD_BACKEND=true
|
||||
# Existing Ollama config remains as fallback
|
||||
- OLLAMA_HOST=http://ollama:11434
|
||||
- OLLAMA_DEFAULT_MODEL=mistral-nemo:latest
|
||||
```
|
||||
|
||||
- [ ] **Configure secrets management**
|
||||
|
||||
Options:
|
||||
1. **Docker secrets** (recommended for Swarm):
|
||||
```yaml
|
||||
secrets:
|
||||
anthropic_api_key:
|
||||
external: true
|
||||
services:
|
||||
tatlock:
|
||||
secrets:
|
||||
- anthropic_api_key
|
||||
environment:
|
||||
- ANTHROPIC_API_KEY_FILE=/run/secrets/anthropic_api_key
|
||||
```
|
||||
|
||||
2. **Environment file** (simpler but less secure):
|
||||
```yaml
|
||||
services:
|
||||
tatlock:
|
||||
env_file:
|
||||
- ./secrets/tatlock.env
|
||||
```
|
||||
|
||||
- [ ] **Update `CONTAINERS.md`**
|
||||
- Document new environment variables
|
||||
- Note Claude as preferred backend with Ollama fallback
|
||||
- Update any API documentation
|
||||
|
||||
### Priority: Medium (Phase 2 - MCP Server)
|
||||
|
||||
- [ ] **Add `tatlock-mcp` service definition** (when Phase 2 is ready)
|
||||
```yaml
|
||||
tatlock-mcp:
|
||||
image: git.schweitz.net/jpmschweitzer/tatlock:latest
|
||||
command: ["python", "-m", "src.mcp.server"]
|
||||
ports:
|
||||
- "8778:8778"
|
||||
environment:
|
||||
- MCP_AUTH_TOKEN=${MCP_AUTH_TOKEN}
|
||||
# ... other config
|
||||
```
|
||||
|
||||
### Priority: Low (Optimization)
|
||||
|
||||
- [ ] **Review resource limits**
|
||||
- Claude backend may have different resource profile than Ollama
|
||||
- Monitor memory/CPU after deployment
|
||||
- Adjust limits if needed
|
||||
|
||||
## Testing
|
||||
|
||||
After stack update:
|
||||
|
||||
```bash
|
||||
# Check Tatlock logs for backend selection
|
||||
docker service logs tatlock_tatlock 2>&1 | grep -E "claude|backend"
|
||||
|
||||
# Should see:
|
||||
# model_backend_configured backend=claude model=claude-sonnet-4-20250514
|
||||
# OR (if no API key):
|
||||
# claude_health_check_skipped reason=no_api_key
|
||||
# model_backend_configured backend=ollama
|
||||
```
|
||||
|
||||
## Timeline
|
||||
|
||||
- **Blocking:** Yes - required for production Claude deployment
|
||||
- **When to implement:** When ANTHROPIC_API_KEY is available
|
||||
- **Effort:** ~1 hour for stack updates, ~30 min for secrets setup
|
||||
Reference in New Issue
Block a user