134 lines
3.6 KiB
Markdown
134 lines
3.6 KiB
Markdown
# GPU Docker Configuration - Working Setup
|
|
|
|
> Successfully configured: 2025-11-11
|
|
> System: tower-of-joy
|
|
> GPU: NVIDIA GeForce RTX 2080 Ti
|
|
> Driver: 470.256.02
|
|
|
|
## Problem Encountered
|
|
|
|
**Issue:** nvidia-container-toolkit 1.18.0 has a compatibility bug with NVIDIA driver 470.x
|
|
- Version 1.18 changed default mode from "legacy" to "CDI"
|
|
- Legacy mode detection fails with older drivers
|
|
- Error: `libnvidia-ml.so.1: cannot open shared object file`
|
|
|
|
## Working Solution
|
|
|
|
**Downgrade to nvidia-container-toolkit 1.17.9-1**
|
|
|
|
All nvidia-container packages must be downgraded together:
|
|
- nvidia-container-toolkit
|
|
- nvidia-container-toolkit-base
|
|
- libnvidia-container-tools
|
|
- libnvidia-container1
|
|
|
|
## Installation Commands
|
|
|
|
```bash
|
|
# Remove all nvidia-container packages
|
|
sudo apt-get remove -y nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1
|
|
|
|
# Clean up
|
|
sudo apt-get autoremove -y
|
|
|
|
# Install all packages at version 1.17.9-1
|
|
sudo apt-get install -y \
|
|
nvidia-container-toolkit=1.17.9-1 \
|
|
nvidia-container-toolkit-base=1.17.9-1 \
|
|
libnvidia-container-tools=1.17.9-1 \
|
|
libnvidia-container1=1.17.9-1
|
|
|
|
# Hold packages to prevent auto-upgrade
|
|
sudo apt-mark hold nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1
|
|
|
|
# Configure Docker runtime
|
|
sudo nvidia-ctk runtime configure --runtime=docker --config=/etc/docker/daemon.json
|
|
|
|
# Rebuild library cache
|
|
sudo ldconfig
|
|
|
|
# Restart Docker
|
|
sudo systemctl restart docker
|
|
```
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
# Test GPU access
|
|
sudo docker run --rm --gpus all nvidia/cuda:11.8.0-runtime-ubuntu20.04 nvidia-smi
|
|
```
|
|
|
|
**Expected output:** nvidia-smi showing RTX 2080 Ti
|
|
|
|
## Current Configuration
|
|
|
|
### /etc/docker/daemon.json
|
|
```json
|
|
{
|
|
"runtimes": {
|
|
"nvidia": {
|
|
"args": [],
|
|
"path": "nvidia-container-runtime"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Installed Versions
|
|
```
|
|
libnvidia-container-tools 1.17.9-1
|
|
libnvidia-container1 1.17.9-1
|
|
nvidia-container-toolkit 1.17.9-1
|
|
nvidia-container-toolkit-base 1.17.9-1
|
|
```
|
|
|
|
All packages are **held** to prevent automatic upgrade to 1.18.x
|
|
|
|
## Important Notes
|
|
|
|
1. **Do NOT upgrade** nvidia-container-toolkit to 1.18.x - it breaks compatibility with driver 470
|
|
2. If you run `apt upgrade`, the packages are held and won't upgrade
|
|
3. To check held packages: `apt-mark showhold`
|
|
4. To unhold (not recommended): `sudo apt-mark unhold nvidia-container-toolkit`
|
|
|
|
## For Future Reference
|
|
|
|
If you need to update the NVIDIA driver:
|
|
1. Driver 470.x is compatible with CUDA 11.4
|
|
2. Driver 525+ is compatible with CUDA 12.x
|
|
3. After driver update, may be able to use newer nvidia-container-toolkit
|
|
|
|
## Testing GPU in Containers
|
|
|
|
### Quick Test
|
|
```bash
|
|
docker run --rm --gpus all nvidia/cuda:11.8.0-runtime-ubuntu20.04 nvidia-smi
|
|
```
|
|
|
|
### Test with Ollama
|
|
```bash
|
|
docker run --rm --gpus all ollama/ollama nvidia-smi
|
|
```
|
|
|
|
### Test with Jellyfin (after deployment)
|
|
Check Jellyfin Dashboard → Playback → Transcoding for NVIDIA NVENC option
|
|
|
|
## Troubleshooting
|
|
|
|
If GPU stops working after system update:
|
|
|
|
```bash
|
|
# Check if packages were upgraded
|
|
dpkg -l | grep nvidia-container
|
|
|
|
# If upgraded to 1.18.x, re-run downgrade script:
|
|
sudo bash /home/jpmschweitzer/Projects/tower-of-joy/scripts/gpu-fix-downgrade-all.sh
|
|
```
|
|
|
|
## References
|
|
|
|
- NVIDIA Container Toolkit: https://github.com/NVIDIA/nvidia-container-toolkit
|
|
- Driver 470 Release Notes: https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-470-256-02/
|
|
- Issue with 1.18.0: https://github.com/NVIDIA/nvidia-container-toolkit/issues
|
|
- Our fix script: `/home/jpmschweitzer/Projects/tower-of-joy/scripts/gpu-fix-downgrade-all.sh`
|