ok... ok... I'll add it to git...

2025-11-14 15:31:25 +01:00
commit 664fe55ff4
106 changed files with 24602 additions and 0 deletions
@@ -0,0 +1,133 @@
+# GPU Docker Configuration - Working Setup
+
+> Successfully configured: 2025-11-11
+> System: tower-of-joy
+> GPU: NVIDIA GeForce RTX 2080 Ti
+> Driver: 470.256.02
+
+## Problem Encountered
+
+**Issue:** nvidia-container-toolkit 1.18.0 has a compatibility bug with NVIDIA driver 470.x
+- Version 1.18 changed default mode from "legacy" to "CDI"
+- Legacy mode detection fails with older drivers
+- Error: `libnvidia-ml.so.1: cannot open shared object file`
+
+## Working Solution
+
+**Downgrade to nvidia-container-toolkit 1.17.9-1**
+
+All nvidia-container packages must be downgraded together:
+- nvidia-container-toolkit
+- nvidia-container-toolkit-base
+- libnvidia-container-tools
+- libnvidia-container1
+
+## Installation Commands
+
+```bash
+# Remove all nvidia-container packages
+sudo apt-get remove -y nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1
+
+# Clean up
+sudo apt-get autoremove -y
+
+# Install all packages at version 1.17.9-1
+sudo apt-get install -y \
+  nvidia-container-toolkit=1.17.9-1 \
+  nvidia-container-toolkit-base=1.17.9-1 \
+  libnvidia-container-tools=1.17.9-1 \
+  libnvidia-container1=1.17.9-1
+
+# Hold packages to prevent auto-upgrade
+sudo apt-mark hold nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1
+
+# Configure Docker runtime
+sudo nvidia-ctk runtime configure --runtime=docker --config=/etc/docker/daemon.json
+
+# Rebuild library cache
+sudo ldconfig
+
+# Restart Docker
+sudo systemctl restart docker
+```
+
+## Verification
+
+```bash
+# Test GPU access
+sudo docker run --rm --gpus all nvidia/cuda:11.8.0-runtime-ubuntu20.04 nvidia-smi
+```
+
+**Expected output:** nvidia-smi showing RTX 2080 Ti
+
+## Current Configuration
+
+### /etc/docker/daemon.json
+```json
+{
+    "runtimes": {
+        "nvidia": {
+            "args": [],
+            "path": "nvidia-container-runtime"
+        }
+    }
+}
+```
+
+### Installed Versions
+```
+libnvidia-container-tools     1.17.9-1
+libnvidia-container1          1.17.9-1
+nvidia-container-toolkit      1.17.9-1
+nvidia-container-toolkit-base 1.17.9-1
+```
+
+All packages are **held** to prevent automatic upgrade to 1.18.x
+
+## Important Notes
+
+1. **Do NOT upgrade** nvidia-container-toolkit to 1.18.x - it breaks compatibility with driver 470
+2. If you run `apt upgrade`, the packages are held and won't upgrade
+3. To check held packages: `apt-mark showhold`
+4. To unhold (not recommended): `sudo apt-mark unhold nvidia-container-toolkit`
+
+## For Future Reference
+
+If you need to update the NVIDIA driver:
+1. Driver 470.x is compatible with CUDA 11.4
+2. Driver 525+ is compatible with CUDA 12.x
+3. After driver update, may be able to use newer nvidia-container-toolkit
+
+## Testing GPU in Containers
+
+### Quick Test
+```bash
+docker run --rm --gpus all nvidia/cuda:11.8.0-runtime-ubuntu20.04 nvidia-smi
+```
+
+### Test with Ollama
+```bash
+docker run --rm --gpus all ollama/ollama nvidia-smi
+```
+
+### Test with Jellyfin (after deployment)
+Check Jellyfin Dashboard → Playback → Transcoding for NVIDIA NVENC option
+
+## Troubleshooting
+
+If GPU stops working after system update:
+
+```bash
+# Check if packages were upgraded
+dpkg -l | grep nvidia-container
+
+# If upgraded to 1.18.x, re-run downgrade script:
+sudo bash /home/jpmschweitzer/Projects/tower-of-joy/scripts/gpu-fix-downgrade-all.sh
+```
+
+## References
+
+- NVIDIA Container Toolkit: https://github.com/NVIDIA/nvidia-container-toolkit
+- Driver 470 Release Notes: https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-470-256-02/
+- Issue with 1.18.0: https://github.com/NVIDIA/nvidia-container-toolkit/issues
+- Our fix script: `/home/jpmschweitzer/Projects/tower-of-joy/scripts/gpu-fix-downgrade-all.sh`