Blindly force-rebooting a multi-GPU node during an active training run
Check whether workloads are holding devices open and capture dmesg telemetry first.
Operator task guide
Driver/library version mismatch occurs when user-space NVIDIA utilities (libnvidia-ml.so and nvidia-smi) upgrade to a newer version (for example 570.x or 580.x) while the older nvidia.ko kernel module remains loaded in RAM.
Always verify running GPU jobs and preserve /proc/driver/nvidia/version and dmesg logs before unloading kernel modules or scheduling a clean node drain and reboot.
Reviewed . Reference guidance is not a diagnosis of your workload.
Confirm the exact split between the resident kernel module and user-space libraries.
cat /proc/driver/nvidia/version
dpkg -l | rg 'nvidia-driver|libnvidia-gl|nvidia-utils' || rpm -qa | rg -i 'nvidia'If no training or inference job is running, you can reload the kernel modules cleanly without rebooting the entire server.
lsmod | rg nvidia
sudo fuser -v /dev/nvidia* || lsof /dev/nvidia*Stop persistence daemon or container runtimes holding module references, reload nvidia and nvidia_uvm, then mask automated unattended upgrades on production GPU clusters.
sudo systemctl stop nvidia-persistenced || true
sudo modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia
sudo modprobe nvidia && nvidia-smi| Signal | What it means | Next action |
|---|---|---|
| Failed to initialize NVML: Driver/library version mismatch | User-space library upgraded past loaded nvidia kernel module | Drain idle jobs, reload nvidia kernel modules or reboot, and hold nvidia-* packages in unattended-upgrades. |
| No devices were found after driver reload | PCIe GPU dropped from bus (check Xid 79) or VFIO/IOMMU bound the device | Run lspci | grep -i nvidia and inspect dmesg for NVRM Xid 79 before treating as driver config. |
| Mismatch inside Docker container only | Container mounted incompatible host driver library paths or nvidia-container-runtime hook failed | Verify host nvidia-smi works, restart containerd/docker, and check CDI specs. |
Check whether workloads are holding devices open and capture dmesg telemetry first.
Ubuntu automatic security updates routinely bump libnvidia-* user-space packages mid-week, breaking nvidia-smi on idle login.
Yes, if nothing is holding /dev/nvidia* device files open (after stopping nvidia-persistenced and containers), unloading nvidia_drm, nvidia_modeset, nvidia_uvm, and nvidia via modprobe -r and reloading nvidia synchronizes user-space and kernel versions without rebooting.
If nvidia-smi loads cleanly and still prints No devices were found, check lspci for physical PCIe enumeration, host dmesg for Xid 79 bus drops, or confirm host GPUs are not bound to vfio-pci.
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident