portainer-core/AGENTS.md

# AGENTS.md

**A README for AI coding agents** - This file provides technical context and guidelines for LLM coding agents working on this project.

> This follows the [agents.md](https://agents.md/) format: a simple, open standard for guiding coding agents across 20+ tools including Claude Code, Cursor, GitHub Copilot, and more.

## About This File

- **Purpose**: Single source of truth for all LLM coding agents
- **Scope**: Technical details, build steps, code conventions, and testing procedures
- **Maintenance**: Update this file (AGENTS.md) only—don't modify agent-specific redirect files (CLAUDE.md, .clinerules, etc.)
- **Living Document**: Keep this updated as project practices evolve

## Project Overview

This is the `tower-of-joy` project - a containerized home server infrastructure for the "tower-of-joy" system.

**Purpose:** Self-hosted services platform with GPU-accelerated ML model serving, media streaming, cloud storage, and secure remote access.

**System:** Intel i7-6700, RTX 2080 Ti (11GB VRAM), 16GB RAM, Zorin OS 16.3 (Ubuntu 20.04 based)

**Storage Architecture:**
- **SSD (489GB):** Container configs, databases, Docker images - `/home/jpmschweitzer/docker-data/`
- **HDD (3.7TB):** User content, media, backups - `/mnt/media/`

**Key Components:**
- **Container Management:** Portainer (port 8080)
- **Reverse Proxy:** Nginx Proxy Manager (port 8000 - unified web interface)
- **ML Models:** Ollama with GPU acceleration (port 11434)
- **Game Servers:** AMP integration (port 8081)
- **Networking:** Headscale/Tailscale for secure remote access
- **Applications (Backlog):** Jellyfin, Nextcloud, Samba file sharing

**Architecture Decision:** Portainer + Docker Compose chosen over full NAS solutions (TrueNAS/Unraid) to avoid OS reinstall and leverage existing Docker installation.

**External Access Rule:** All public/external services MUST route through Nginx Proxy Manager (NPM) for Let's Encrypt SSL management and unified logging. Never expose service ports directly to the internet (except NPM and Headscale).

**Service Maintenance**
- **Stack Management** the portainer (and docker) and NPM services are managed through the core-api service. To maintain settings and configurations of these systems, read the documentation at http://tower-of-joy:8083/docs and prefer to use the api functions over direct reads an edits.

**Service Integration Policy:** A service deployment is INCOMPLETE until cross-service integrations are implemented. Every new service MUST be integrated with:
- **Tatlock UI:** Add service to dashboard configuration
- **docs/reference/CONTAINERS.md:** Document the service with full profile and configuration table

Services without dashboard integration are considered unfinished and should not be marked as "complete" in STATUS.md or commit messages.

## Build & Run Commands

This project uses Docker Compose via Portainer for service management.

### Infrastructure Management

```bash
# Check all running containers
docker ps

# View specific service logs
docker logs <container-name>

# Restart a service
docker restart <container-name>

# Deploy a stack from /stacks directory
docker stack deploy -c stacks/<stack-name>.yml <stack-name>

# Clean up unused resources (run monthly)
docker system prune -a
```

### GPU Verification

```bash
# Verify NVIDIA driver
nvidia-smi

# Test GPU access in Docker
docker run --rm --gpus all nvidia/cuda:11.4.0-base-ubuntu20.04 nvidia-smi

# Monitor GPU during ML inference or transcoding
watch -n 1 nvidia-smi
```

### Storage Management

```bash
# Check disk usage
df -h

# SSD usage (configs)
du -sh ~/docker-data/*

# HDD usage (content)
du -sh /mnt/media/*

# Verify media drive mounted
mount | grep /mnt/media
```

### Service Access

```bash
# All services accessible via browser:
# Portainer:     http://localhost:8080
# NPM (unified): http://localhost:8000
# AMP (games):   http://localhost:8081
# Ollama API:    http://localhost:11434
```

## Testing

### Infrastructure Testing

```bash
# Test GPU passthrough in containers
docker run --rm --gpus all nvidia/cuda:11.4.0-base-ubuntu20.04 nvidia-smi

# Test Ollama GPU acceleration
docker exec ollama nvidia-smi
docker exec ollama ollama run llama3.2:3b "Hello, test response"

# Verify media drive accessibility
touch /mnt/media/test-write.txt && rm /mnt/media/test-write.txt

# Test network connectivity to services
curl -I http://localhost:8080  # Portainer
curl -I http://localhost:8000  # NPM
curl http://localhost:11434/api/tags  # Ollama models list

# Test Headscale/Tailscale mesh (if configured)
tailscale status
ping <tailscale-ip>
```

### Testing Guidelines
- **Before deploying new stacks:** Test GPU access if service needs GPU
- **After configuration changes:** Verify service still responds on expected ports
- **Storage changes:** Test read/write permissions on both SSD and HDD paths
- **Network changes:** Ensure Headscale connectivity maintained
- **GPU workloads:** Monitor `nvidia-smi` to confirm GPU utilization during transcoding/inference

## Code Style & Conventions

### General Principles
- Follow existing code formatting and conventions in the codebase
- Use meaningful variable and function names
- Add comments for complex logic
- Prefer clarity over cleverness

### Specific Conventions

**Docker Compose:**
- Use `version: '3.8'` for all compose files
- Always include `restart: unless-stopped` for production services
- Use meaningful container names: `container_name: service-name`
- Network isolation: Create dedicated networks per stack
- Volume paths: Use full absolute paths, never relative paths

**Storage Convention:**
- **SSD paths:** `/home/jpmschweitzer/docker-data/<service>/` - for configs, cache, databases
- **HDD paths:** `/mnt/media/<service>/` - for user content, media, bulk data
- Always document in compose file which disk each volume uses

**GPU Services:**
- Include NVIDIA environment variables:
  ```yaml
  environment:
    - NVIDIA_VISIBLE_DEVICES=all
    - NVIDIA_DRIVER_CAPABILITIES=all
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
  ```

**Port Allocation:**
- Document all port assignments in /stacks/README.md
- Avoid port conflicts with AMP game servers (varies by instance)
- Reserve 8000-8099 range for infrastructure services

## Security Guidelines

**Security first** - Never introduce vulnerabilities:
- No SQL injection vulnerabilities
- No XSS vulnerabilities
- No command injection vulnerabilities
- No hardcoded credentials or API keys
- Follow OWASP Top 10 best practices

### Sensitive Information
- Never commit credentials, API keys, or secrets
- Use environment variables for sensitive configuration
- Review changes before committing to catch accidental exposure

## Git Workflow

### Commits

Use conventional commit format for infrastructure changes:

**Format:** `<type>(<scope>): <description>`

**Types:**
- `feat`: New service or capability (e.g., "feat(stack): add nginx proxy manager")
- `fix`: Fix broken service or configuration
- `config`: Configuration changes to existing services
- `docs`: Documentation updates
- `chore`: Maintenance tasks (cleanup, updates)

**Examples:**
```bash
feat(stack): add ollama with GPU support for ML inference
fix(portainer): correct volume mount permissions for config
config(nginx): update proxy host for new service
docs(readme): add troubleshooting section for GPU passthrough
chore(cleanup): remove unused docker volumes
```

**Scope Guidelines:**
- Use service name for single-service changes: `(ollama)`, `(jellyfin)`
- Use `(stack)` for multi-service deployments
- Use `(storage)` for disk/volume changes
- Use `(network)` for Headscale/networking changes

### Pull Requests

**Before Creating PR:**
- [ ] Test service deployment: `docker ps` shows container running
- [ ] Verify GPU access (if applicable): `docker exec <container> nvidia-smi`
- [ ] Check service responds: `curl -I http://localhost:<port>`
- [ ] Update /stacks/README.md with new service ports
- [ ] Document storage paths used (SSD vs HDD)
- [ ] Update STATUS.md if completing a phase

**PR Description Template:**
```markdown
## Changes
- Service deployed: [service name]
- Ports used: [list ports]
- GPU required: [yes/no]

## Storage
- SSD: [path to configs]
- HDD: [path to data, if applicable]

## Testing
- [ ] Container starts successfully
- [ ] Service accessible at expected port
- [ ] GPU passthrough verified (if needed)
- [ ] Integration with existing services tested

## Documentation
- [ ] Added to /stacks/README.md
- [ ] Updated STATUS.md
- [ ] Added troubleshooting notes if needed
```

### Gitignore Discipline

**CRITICAL: Never commit ephemeral or local files to the repository.**

Always maintain proper .gitignore hygiene:

#### Files to NEVER Commit
- **Build artifacts**: `dist/`, `build/`, `*.o`, `*.pyc`, compiled binaries
- **Dependencies**: `node_modules/`, `vendor/`, `venv/`, `.pnp.*`
- **IDE/Editor files**: `.vscode/`, `.idea/`, `*.swp`, `.DS_Store`
- **OS files**: `Thumbs.db`, `.DS_Store`, `desktop.ini`
- **Log files**: `*.log`, `logs/`, `npm-debug.log*`
- **Environment files**: `.env`, `.env.local`, `.env.*.local`
- **Cache directories**: `.cache/`, `.pytest_cache/`, `__pycache__/`
- **Coverage reports**: `coverage/`, `.coverage`, `*.lcov`
- **Temporary files**: `*.tmp`, `*.temp`, `.tmp/`, scratch files

#### Before Committing
1. **Check git status** - Review all files being added
2. **Verify .gitignore** - Ensure appropriate patterns are present
3. **Update .gitignore** - Add new patterns for any generated/local files discovered
4. **Never use `git add .` blindly** - Be intentional about what gets staged

#### When Adding New Tools/Dependencies
- Immediately update .gitignore with appropriate patterns
- Check the tool's documentation for recommended ignore patterns
- Use gitignore.io or templates for common patterns

#### Red Flags
If you see any of these in `git status`, STOP and add to .gitignore:
- Hundreds of files in a dependency directory
- Binary files (unless intentionally tracked)
- Files with local machine paths
- Auto-generated documentation
- Test output or artifacts

## Documentation

- Keep README files up to date
- Update relevant documentation when changing functionality
- Document complex algorithms and business logic
- Maintain API documentation if applicable

## Agent Guidelines

### Before Making Changes
1. **Read this file first** - Understand project context and conventions
2. **Explore the codebase** - Understand existing patterns before adding new code
3. **Ask questions** - Clarify requirements before major changes

### Prompt Logging
**MANDATORY: Log all user prompts before starting work.**

When a user provides a prompt/request:
1. **Immediately log it** to `prompt-log.md` in the project root
2. **Include timestamp** - Use ISO 8601 format (YYYY-MM-DD HH:MM:SS)
3. **Preserve exact wording** - Copy the user's prompt verbatim
4. **Then begin work** - After logging, proceed with the task

This ensures no work is lost if sessions are accidentally closed.

**Format:**
```markdown
## [YYYY-MM-DD HH:MM:SS]

[User's exact prompt here]

---
```

### Working Principles
- **Specificity over generality**: Follow exact commands and patterns
- **Consistency**: Match existing code style and architecture
- **Communication**: Explain what you're doing and why
- **Verification**: Test changes before marking tasks complete

### Service Deployment Checklist

When deploying a NEW service, follow this complete checklist. A deployment is **INCOMPLETE** until all steps are finished:

**Phase 1: Container Deployment**
- [ ] Create Docker Compose stack in `stacks/` directory
- [ ] Configure volumes (SSD for configs, HDD for data)
- [ ] Set appropriate resource limits
- [ ] Configure GPU if needed (use deploy.resources.reservations)
- [ ] Set restart policy to `unless-stopped`
- [ ] Deploy via Portainer
- [ ] Verify container is running: `docker ps | grep <service>`
- [ ] Check logs for errors: `docker logs <service>`

**Phase 2: Service Configuration**
- [ ] Complete initial service setup wizard (if applicable)
- [ ] Configure service-specific settings
- [ ] Generate API keys/tokens if needed
- [ ] Test service accessibility at designated port
- [ ] Document access credentials securely

**Phase 3: Cross-Service Integration (MANDATORY)**
- [ ] **Tatlock UI Integration:**
  - Add service to dashboard configuration
  - Test service appears correctly in dashboard
- [ ] **NPM Integration (if externally accessible):**
  - Create proxy host entry
  - Configure SSL with Let's Encrypt
  - Test external access through proxy

**Phase 4: Documentation (MANDATORY)**
- [ ] Add service profile to `docs/reference/CONTAINERS.md` with:
  - One-paragraph description
  - Complete configuration table
  - All dependencies listed
- [ ] Add service to `docs/reference/CONTAINERS.md` quick reference tables:
  - Service Access Matrix
  - Storage Distribution (if uses storage)
  - GPU-Enabled Services (if uses GPU)
- [ ] Update `STATUS.md` to reflect new service deployment
- [ ] Update `README.md` service ports table if needed

**Phase 5: Verification**
- [ ] Service accessible at documented URL
- [ ] Docker healthcheck shows healthy status
- [ ] Tatlock UI displays service correctly
- [ ] Service persists across container restart
- [ ] Backups configured (if service has important data)

**Example of COMPLETE deployment:**
```bash
# 1. Deploy Jellyfin container
# 2. Configure Jellyfin settings and add media
# 3. Add Jellyfin to Tatlock UI dashboard
# 4. Document in CONTAINERS.md
# 5. Test all integrations work
# ✓ NOW the deployment is complete
```

**DO NOT** mark a service as "deployed" or "complete" in STATUS.md or commit messages until ALL checklist items are finished, especially cross-service integrations.

### Sudo and Privileged Commands
**CRITICAL: LLM agents CANNOT execute sudo commands.**

- **NEVER attempt to run sudo commands** - They will always fail due to password requirements
- **NEVER retry sudo commands repeatedly** - If a sudo command fails, don't try variations
- **Instead: Provide clear instructions** - Give the user the exact commands to run in their terminal
- **Format commands clearly** - Use markdown code blocks with explanations
- **Wait for user confirmation** - After providing sudo commands, wait for the user to confirm they've run them
- **Verify results** - After the user runs commands, check the results with non-privileged commands

**Example Workflow:**
1. Detect that a task requires sudo (e.g., system services, file permissions, package installation)
2. Provide the user with clear, formatted commands to run
3. Explain what each command does
4. Wait for user to report results
5. Verify with non-sudo commands (e.g., check service status, file existence)

## Monorepo Instructions

If this project grows into a monorepo, place nested AGENTS.md files in subpackages with package-specific instructions. Agents will read the closest AGENTS.md in the directory tree.

---

*Format based on [agents.md](https://agents.md/) - Last updated: 2025-11-11*