Operator task guide

How to diagnose silent GPU stragglers

In distributed training, silent stragglers force all healthy GPUs to spin in busy-wait barrier loops while nvidia-smi still reports 100 percent utilization.

Profile per-rank forward and backward step duration before the all-reduce collective. Check DCGM SM clock throttling, fan tachometers, and PCIe replay counters on the slowest reporting rank.

Step-by-step method

1
Isolate step time variance before the collective barrier
Measure the elapsed time per rank for compute versus communication. If all ranks show long collective durations but only one rank has a prolonged forward or backward compute phase, that rank is your straggler.
```
export TORCH_DISTRIBUTED_DEBUG=DETAIL
python -m torch.distributed.run --nproc_per_node=8 train.py
```
2
Check physical GPU clock frequencies and thermal state
A failed cooling fan or dirty heat sink causes thermal throttling. The GPU downclocks its SM from nominal boost clocks to base or throttle clocks, dragging the entire cluster down to its speed.
```
nvidia-smi --query-gpu=index,name,clocks.current.sm,clocks.max.sm,temperature.gpu,fan.speed --format=csv
nvidia-smi -q -d PERFORMANCE
```
3
Inspect PCIe link degradation and replay counters
A marginal PCIe riser or dirty gold finger connector can cause a link to negotiate at Gen1 speeds instead of Gen5, or accumulate thousands of replayed packets during weight transfers.
```
lspci -vvv -s <pci-bus-id> | rg -i 'LnkSta:'
nvidia-smi nvlink --errors
```
4
Drain the straggler node and resume execution
Once the physical device or host is identified, drain the node from the cluster scheduler and resume the job from the latest clean checkpoint excluding the defective node.
```
scontrol update NodeName=<straggler-node> State=DRAIN Reason='Denpex: thermal throttle straggler'
sbatch --exclude=<straggler-node> run_job.slurm
```

Straggler symptom decision table

Signals, meanings and actions for how to diagnose silent gpu stragglers.
Signal	What it means	Next action
SM clock drops under 1,200 MHz under load	GPU thermal throttling or power brake active	Inspect chassis airflow, fan tachometer, and power limit capping.
PCIe link running at Gen1 x16 instead of Gen5 x16	Physical bus degradation or marginal riser seating	Reseat GPU and inspect gold fingers or schedule node RMA swap.
Long ncclAllReduce duration on 511 of 512 GPUs	511 victim GPUs are busy-waiting for 1 slow rank	Trace step time backward to find the single rank with long compute duration.
High NVLink CRC error replay count	Degraded high-speed interconnect link	Inspect ibdiagnet or DCGM counters and drain affected compute node.

Evidence checklist

Per-rank compute step time and barrier wait duration
nvidia-smi clock frequencies and temperature logs
PCIe negotiated link speed and capability from lspci
DCGM throttle violation counters and fan speeds
Host dmesg hardware error events during step slowdown

Common mistakes

Trusting 100 percent GPU utilization graphs

GPUs spinning in a NCCL barrier busy-wait loop report 100 percent utilization in nvidia-smi while doing zero productive work.

Lowering the global batch size

A hardware-throttled straggler will still slow down the entire cluster regardless of micro-batch size adjustments.

Blaming network congestion without inspecting clocks

Slow all-reduce barriers are often caused by one slow GPU completing its backward pass late, not by the network fabric.

Tools and deeper references

GPU reliability overview

Understand failure distributions and silent performance degradations.

First failed rank guide

Identify the initiating rank in cascading timeout incidents.

GPU downtime calculator

Calculate the exact capital cost of cluster slowdowns.

Denpex vs Datadog

Learn why metric aggregations miss per-rank barrier stragglers.

Official references

Frequently asked questions

Why does nvidia-smi show 100 percent utilization on all GPUs when a straggler exists?

PyTorch distributed collectives use spin-locks to wait for lagging ranks. The GPU cores remain fully active polling memory addresses, drawing high power and reporting 100 percent utilization while completely stalled.

How much does a single straggler cost a large cluster?

Because distributed training synchronizes at every step, one GPU running 40 percent slower slows the entire cluster by 40 percent. On a 512x H100 cluster, this wastes over 700 dollars per hour in burned compute.

Can a network cable cause a GPU straggler?

Yes. An InfiniBand or RoCE cable with high optical symbol errors will trigger frequent packet retransmissions and link retraining, delaying data arrival on that node and forcing all peer ranks to wait.

Apply the method to the complete incident

Paste the logs for an answer-first diagnosis, evidence request and verification step.

Diagnose the incident