Operator task guide

How to diagnose multi-node cluster drift

When identical containers succeed on Node 0 but fail on Node 14, compare host-level kernel, driver, topology, and PCIe configurations between the golden and defective node.

Isolate subtle divergences such as driver minor versions, PCIe link generation downgrade, CPU NUMA socket memory binding, and sysctl network buffer settings.

Step-by-step method

1
Compare NVIDIA driver and kernel modules across nodes
Unattended OS upgrades or partial node reboots frequently leave different hosts running mismatched kernel modules or library ABIs, breaking inter-node collectives.
```
nvidia-smi --query-gpu=driver_version --format=csv,noheader
cat /proc/driver/nvidia/version
```
2
Audit PCIe link speed and generation negotiated
Verify that every GPU on both nodes has negotiated its maximum bus speed (such as Gen5 x16). A degraded riser or marginal lane drops bandwidth to Gen3 or Gen1.
```
lspci -vvv | rg -E '(LnkCap|LnkSta):.*Speed.*Width'
```
3
Check CPU NUMA node and socket affinity
If a GPU on one node is bound to a remote CPU socket across the UPI/QPI interconnect rather than its local NUMA node, host-to-device memory transfers suffer severe latency.
```
nvidia-smi topo -m
numactl --hardware
```
4
Verify kernel sysctl and network buffer settings
Distributed training requires expanded network receive and transmit buffers. Ensure host sysctls match across all nodes in the partition.
```
sysctl net.core.rmem_max net.core.wmem_max net.ipv4.tcp_rmem
sysctl vm.max_map_count
```

Configuration divergence decision table

Signals, meanings and actions for how to diagnose multi-node cluster drift.
Signal	What it means	Next action
Driver version 535.129 on Node 14 vs 535.154 on Node 0	Driver minor version mismatch causing collective transport fallback	Synchronize NVIDIA driver and CUDA user-space versions across the fleet.
PCIe link speed 8GT/s (Gen3) instead of 32GT/s (Gen5)	Hardware link negotiation defect on the host motherboard	Reseat GPU card or mark host for hardware vendor maintenance.
NUMA affinity crossing CPU sockets on one host	Process launcher failed to pin processes to local NUMA node	Configure numactl or framework launcher with local socket binding.
Container /dev/shm default 64MB on one host	Docker or Kubernetes container runtime configuration variance	Mount dshm volume with sufficient shared memory for PyTorch collectives.

Evidence checklist

NVIDIA driver version and CUDA runtime build on both nodes
lspci link status and capabilities for all GPU PCI addresses
nvidia-smi topo matrix showing NVLink and PCIe connections
Host sysctl buffer and memory limits
Scheduler container mount flags and shared memory sizes

Common mistakes

Assuming identical Docker images guarantee identical environments

Docker containers share the host Linux kernel, NVIDIA kernel modules, and PCIe hardware topology, which vary across hosts.

Testing nodes in isolation without a golden peer

Single-node tests often pass because they do not stress multi-node network buffers, cross-socket UPI links, or inter-host collectives.

Ignoring minor driver patch version differences

NCCL transport selections can fall back from NVLink to shared memory when peer nodes run divergent driver releases.

Tools and deeper references

Cross-run environment diff tool

Compare cluster environments and configuration logs in your browser.

NCCL environment generator

Generate unified communication variables across heterogeneous hosts.

Preflight cluster verification

Validate topology, PCIe speeds, and bandwidth before launching jobs.

GPU cloud reliability solutions

Maintain configuration consistency across cloud rental fleets.

Official references

Frequently asked questions

Why does my training job fail when all nodes use the same Docker container?

Containers share the underlying host kernel, driver modules, and hardware bus topologies. If one host has a degraded PCIe slot, mismatched driver patch, or improper NUMA pinning, the containerized workload fails despite identical user code.

How does PCIe link downgrade affect multi-node training?

In multi-node training, gradients must be transferred from GPU memory to host NICs. If a GPU drops from Gen5 to Gen1 or Gen3, host-to-device transfers bottleneck the entire global all-reduce communication step.

What is the fastest way to detect node divergence?

Run a peer configuration diff between a working node and the failing node. Compare lspci link capabilities, driver versions, and network sysctls to isolate the divergent hardware or operating system setting in minutes.

Apply the method to the complete incident

Paste the logs for an answer-first diagnosis, evidence request and verification step.

Diagnose the incident