Skip to content

Operator task guide

Fix nvidia-smi Failed to initialize NVML: Driver/library version mismatch

Driver/library version mismatch occurs when user-space NVIDIA utilities (libnvidia-ml.so and nvidia-smi) upgrade to a newer version (for example 570.x or 580.x) while the older nvidia.ko kernel module remains loaded in RAM.

Always verify running GPU jobs and preserve /proc/driver/nvidia/version and dmesg logs before unloading kernel modules or scheduling a clean node drain and reboot.

Reviewed . Reference guidance is not a diagnosis of your workload.

Step-by-step method

  1. 1

    Compare loaded kernel module version against installed packages

    Confirm the exact split between the resident kernel module and user-space libraries.

    cat /proc/driver/nvidia/version
    dpkg -l | rg 'nvidia-driver|libnvidia-gl|nvidia-utils' || rpm -qa | rg -i 'nvidia'
  2. 2

    Check if any active compute process holds /dev/nvidia* open

    If no training or inference job is running, you can reload the kernel modules cleanly without rebooting the entire server.

    lsmod | rg nvidia
    sudo fuser -v /dev/nvidia* || lsof /dev/nvidia*
  3. 3

    Reload kernel modules cleanly or pin apt/dnf unattended-upgrades

    Stop persistence daemon or container runtimes holding module references, reload nvidia and nvidia_uvm, then mask automated unattended upgrades on production GPU clusters.

    sudo systemctl stop nvidia-persistenced || true
    sudo modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia
    sudo modprobe nvidia && nvidia-smi

NVML mismatch and missing device recovery matrix

Signals, meanings and actions for fix nvidia-smi failed to initialize nvml: driver/library version mismatch.
SignalWhat it meansNext action
Failed to initialize NVML: Driver/library version mismatchUser-space library upgraded past loaded nvidia kernel moduleDrain idle jobs, reload nvidia kernel modules or reboot, and hold nvidia-* packages in unattended-upgrades.
No devices were found after driver reloadPCIe GPU dropped from bus (check Xid 79) or VFIO/IOMMU bound the deviceRun lspci | grep -i nvidia and inspect dmesg for NVRM Xid 79 before treating as driver config.
Mismatch inside Docker container onlyContainer mounted incompatible host driver library paths or nvidia-container-runtime hook failedVerify host nvidia-smi works, restart containerd/docker, and check CDI specs.

Evidence checklist

  • Output of cat /proc/driver/nvidia/version
  • Installed nvidia-driver / libnvidia-ml package versions
  • Active holders on /dev/nvidia* from fuser or lsof
  • Host dmesg lines for NVRM or Xid events

Common mistakes

Blindly force-rebooting a multi-GPU node during an active training run

Check whether workloads are holding devices open and capture dmesg telemetry first.

Leaving unattended-upgrades enabled on GPU nodes

Ubuntu automatic security updates routinely bump libnvidia-* user-space packages mid-week, breaking nvidia-smi on idle login.

Frequently asked questions

Can I fix Driver/library version mismatch without rebooting Linux?

Yes, if nothing is holding /dev/nvidia* device files open (after stopping nvidia-persistenced and containers), unloading nvidia_drm, nvidia_modeset, nvidia_uvm, and nvidia via modprobe -r and reloading nvidia synchronizes user-space and kernel versions without rebooting.

Why does No devices were found appear instead of a version mismatch?

If nvidia-smi loads cleanly and still prints No devices were found, check lspci for physical PCIe enumeration, host dmesg for Xid 79 bus drops, or confirm host GPUs are not bound to vfio-pci.

Apply the method to your incident

Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.

Diagnose your incident