NCCL Collective Operation Timeout

A NCCL collective operation failed to complete within the watchdog timeout period because one or more ranks stopped participating. This is the most common distributed training failure and is almost always a downstream symptom of a different root cause (GPU crash, OOM, network loss, or host failure).

NCCL Collective Operation Timeout is a Distributed Communication failure seen during ML training runs. A NCCL collective operation failed to complete within the watchdog timeout period because one or more ranks stopped participating. This is the most common distributed training failure and is almost always a downstream symptom of a different root cause (GPU crash, OOM, network loss, or host failure). Common tags: Nccl, Timeout, Distributed, Watchdog.

The timeout is a symptom, not a root cause. The initiating fault is elsewhere: GPU crash (OOM, Xid), host OOM killer, network link degradation, or storage stall. NCCL watchdog timeout default (30s) may be too short for very large all-reduces across many nodes. Network congestion or PFC/ECN misconfiguration on RoCE fabric causes RDMA retransmission delays exceeding timeout. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• One or more NCCL collectives (all-reduce, all-gather, reduce-scatter) fail to complete
• Watchdog timer fires after the configured timeout period
• Training job aborts with NCCL timeout errors affecting multiple ranks

• Any distributed training job using NCCL for GPU-to-GPU communication
• Multi-node training with 8+ GPUs across InfiniBand or RoCE
• Training jobs with large model parallelism or tensor parallelism

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: NCCL timeout or Watchdog caught collective operation timeout in logs
• ✓Verified signal present: NCCL WARN messages with specific rank and collective name
• ✓Verified signal present: Multiple ranks report timeout simultaneously (cascade pattern)
• ✓Verified signal present: Job exits with ncclInternalError or ncclSystemError
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.

• The timeout is a symptom, not a root cause. The initiating fault is elsewhere: GPU crash (OOM, Xid), host OOM killer, network link degradation, or storage stall.
• NCCL watchdog timeout default (30s) may be too short for very large all-reduces across many nodes
• Network congestion or PFC/ECN misconfiguration on RoCE fabric causes RDMA retransmission delays exceeding timeout