# GPU Docker Configuration - Working Setup

> Successfully configured: 2025-11-11
> System: tower-of-joy
> GPU: NVIDIA GeForce RTX 2080 Ti
> Driver: 470.256.02

## Problem Encountered

**Issue:** nvidia-container-toolkit 1.18.0 has a compatibility bug with NVIDIA driver 470.x
- Version 1.18 changed default mode from "legacy" to "CDI"
- Legacy mode detection fails with older drivers
- Error: `libnvidia-ml.so.1: cannot open shared object file`

## Working Solution

**Downgrade to nvidia-container-toolkit 1.17.9-1**

All nvidia-container packages must be downgraded together:
- nvidia-container-toolkit
- nvidia-container-toolkit-base
- libnvidia-container-tools
- libnvidia-container1

## Installation Commands

```bash
# Remove all nvidia-container packages
sudo apt-get remove -y nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1

# Clean up
sudo apt-get autoremove -y

# Install all packages at version 1.17.9-1
sudo apt-get install -y \
  nvidia-container-toolkit=1.17.9-1 \
  nvidia-container-toolkit-base=1.17.9-1 \
  libnvidia-container-tools=1.17.9-1 \
  libnvidia-container1=1.17.9-1

# Hold packages to prevent auto-upgrade
sudo apt-mark hold nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1

# Configure Docker runtime
sudo nvidia-ctk runtime configure --runtime=docker --config=/etc/docker/daemon.json

# Rebuild library cache
sudo ldconfig

# Restart Docker
sudo systemctl restart docker
```

## Verification

```bash
# Test GPU access
sudo docker run --rm --gpus all nvidia/cuda:11.8.0-runtime-ubuntu20.04 nvidia-smi
```

**Expected output:** nvidia-smi showing RTX 2080 Ti

## Current Configuration

### /etc/docker/daemon.json
```json
{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}
```

### Installed Versions
```
libnvidia-container-tools     1.17.9-1
libnvidia-container1          1.17.9-1
nvidia-container-toolkit      1.17.9-1
nvidia-container-toolkit-base 1.17.9-1
```

All packages are **held** to prevent automatic upgrade to 1.18.x

## Important Notes

1. **Do NOT upgrade** nvidia-container-toolkit to 1.18.x - it breaks compatibility with driver 470
2. If you run `apt upgrade`, the packages are held and won't upgrade
3. To check held packages: `apt-mark showhold`
4. To unhold (not recommended): `sudo apt-mark unhold nvidia-container-toolkit`

## For Future Reference

If you need to update the NVIDIA driver:
1. Driver 470.x is compatible with CUDA 11.4
2. Driver 525+ is compatible with CUDA 12.x
3. After driver update, may be able to use newer nvidia-container-toolkit

## Testing GPU in Containers

### Quick Test
```bash
docker run --rm --gpus all nvidia/cuda:11.8.0-runtime-ubuntu20.04 nvidia-smi
```

### Test with Ollama
```bash
docker run --rm --gpus all ollama/ollama nvidia-smi
```

### Test with Jellyfin (after deployment)
Check Jellyfin Dashboard → Playback → Transcoding for NVIDIA NVENC option

## Troubleshooting

If GPU stops working after system update:

```bash
# Check if packages were upgraded
dpkg -l | grep nvidia-container

# If upgraded to 1.18.x, re-run downgrade script:
sudo bash /home/jpmschweitzer/Projects/tower-of-joy/scripts/gpu-fix-downgrade-all.sh
```

## References

- NVIDIA Container Toolkit: https://github.com/NVIDIA/nvidia-container-toolkit
- Driver 470 Release Notes: https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-470-256-02/
- Issue with 1.18.0: https://github.com/NVIDIA/nvidia-container-toolkit/issues
- Our fix script: `/home/jpmschweitzer/Projects/tower-of-joy/scripts/gpu-fix-downgrade-all.sh`