Straggler / Slow-Rank Detection

One rank runs slower than the rest, so every collective waits on it. There is no crash and no error. Just falling throughput and idle GPUs. Stragglers are the dominant cause of silent efficiency loss at scale and must be detected by comparing per-rank step times, not by waiting for a timeout.

Root cause

A degraded GPU (thermal throttle, ECC row-remap, reduced clocks) on one rank. Slower network path (oversubscribed link, bad cable, PFC pause storms) to one node. CPU/dataloader contention starving one rank's input pipeline.

Recommended fix

Instrument per-rank step time and flag any rank > 1.5x the median (this is what Denpex straggler detection does)

How Denpex helps

Denpex investigates Straggler / Slow-Rank Detection using the evidence you provide or your connected workload collects. Earlier rank, host or application evidence is needed to distinguish an initiating failure from a downstream report.

Straggler / Slow-Rank Detection is a Reliability failure seen during ML training runs. One rank runs slower than the rest, so every collective waits on it. There is no crash and no error. Just falling throughput and idle GPUs. Stragglers are the dominant cause of silent efficiency loss at scale and must be detected by comparing per-rank step times, not by waiting for a timeout. Common tags: Straggler, Slow Rank, Reliability, Distributed.

A degraded GPU (thermal throttle, ECC row-remap, reduced clocks) on one rank. Slower network path (oversubscribed link, bad cable, PFC pause storms) to one node. CPU/dataloader contention starving one rank's input pipeline. NUMA / PCIe misconfiguration on a single host. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Throughput drops with no error in the logs
• All ranks block at all-reduce waiting on one slow rank
• Step time is dominated by the slowest rank

• Large multi-node data/tensor-parallel training
• Heterogeneous nodes or mixed GPU SKUs
• Shared clusters with noisy neighbors
• Jobs sensitive to a single thermally-throttled or ECC-degraded GPU

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: One rank's step time is consistently 1.5-10x the median
• ✓Verified signal present: GPU utilization on most ranks oscillates between 100% and 0% (waiting)
• ✓Verified signal present: MFU / tokens-per-second degrades over hours without a config change
• ✓Verified signal present: No NCCL timeout fires because the slow rank still responds, just late
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

• A degraded GPU (thermal throttle, ECC row-remap, reduced clocks) on one rank
• Slower network path (oversubscribed link, bad cable, PFC pause storms) to one node
• CPU/dataloader contention starving one rank's input pipeline
• NUMA / PCIe misconfiguration on a single host

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.