Finding the Culprit Rank in an NCCL Hang

When one rank stalls, NCCL's synchronous collectives freeze every rank, masking which GPU actually failed. Localizing the culprit needs divergence signals, not the NCCL stack trace.

Finding the Culprit Rank in an NCCL Hang is a Distributed Training failure seen during ML training runs. When one rank stalls, NCCL's synchronous collectives freeze every rank, masking which GPU actually failed. Localizing the culprit needs divergence signals, not the NCCL stack trace. Common tags: Nccl, Hang, Straggler, Flight Recorder.

One rank stopped progressing (data stall, slow I/O, faulted GPU). Synchronous collective semantics block all peers waiting on it. The faulted rank is in a different state than the rest. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• The whole job hangs with no error
• Every rank shows the same NCCL wait, so the stack is unhelpful
• Need to find which specific GPU/rank diverged

• Large model+data parallel jobs with many comm groups
• Long multi-node runs
• Mixed-speed or partially faulted hardware

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: GPU utilization 100% but power draw low (~70W) on the stuck rank
• ✓Verified signal present: All ranks blocked in the same collective (AllReduce/ReduceScatter)
• ✓Verified signal present: No NCCL error emitted, watchdog eventually times out
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.

• One rank stopped progressing (data stall, slow I/O, faulted GPU)
• Synchronous collective semantics block all peers waiting on it
• The faulted rank is in a different state than the rest