NCCL is the library PyTorch, DeepSpeed and Megatron use to move tensors between GPUs. When a distributed job dies, NCCL is usually what prints the error, which is why almost every multi-GPU failure looks like an NCCL failure.
It usually is not. NCCL sits on top of CUDA, the network and the GPUs themselves, so a fault in any of those surfaces here first. The single most useful habit is to ask what NCCL was waiting for, rather than what NCCL said.
The other rule worth internalising: a collective timeout names the ranks that were WAITING, not the rank that failed. The rank that died is silent, because it is dead. Look for the earliest distinct error across all ranks, not the loudest one.
A system call NCCL relies on failed, and NCCL has no more specific code for it. Almost always the transport layer: a socket it could not open, an RDMA device it could not use, or shared memory it could not map.
First action: Re-run with NCCL_DEBUG=INFO. The unhandled error is a summary; the INFO lines immediately before it name the transport and the actual errno, which is the real message.
A collective did not complete within the timeout, so the watchdog aborted the process. This fires on every rank that was WAITING. The rank that failed to arrive prints nothing, because it has already died or hung.
First action: Do not debug the rank that reported this. Set TORCH_NCCL_TRACE_BUFFER_SIZE=2000 to dump which collective each rank was in, then take the earliest distinct error across all rank logs.
ncclUnhandledCudaError: Call to CUDA function failed
A CUDA call inside NCCL failed. NCCL is the messenger; the fault is in CUDA, commonly an earlier asynchronous kernel error that only surfaced when NCCL synchronised.
First action: Set CUDA_LAUNCH_BLOCKING=1 and re-run. Asynchronous CUDA errors are reported at the next sync point, so the traceback normally blames the wrong line until you make launches synchronous.
Internal check failed
Ambiguous, needs evidence
ncclInternalError: Internal check failed
An NCCL invariant did not hold. Often a topology NCCL could not make sense of, or a version mismatch between the NCCL a framework was built against and the one actually loaded.
First action: Confirm which NCCL is loaded: python -c "import torch; print(torch.cuda.nccl.version())" and compare with ldconfig -p | grep nccl. A second NCCL on the path is a common cause.
System call failed
Environment / runtime
ncclSystemError: System call (e.g. socket, malloc) failed
A socket, allocation or file operation failed underneath NCCL. In containers this is usually a limit rather than a bug: locked memory, open file descriptors, or shared memory size.
First action: Check the limits inside the container, not the host: ulimit -l (must be unlimited for RDMA), ulimit -n, and df -h /dev/shm.
Bootstrap interface not found
Configuration
NCCL WARN Bootstrap : no socket interface found
NCCL could not pick an interface to do its initial rendezvous on. Its auto-selection takes the first non-loopback interface, which in a container is often a bridge that does not route between nodes.
First action: Set NCCL_SOCKET_IFNAME explicitly to the fabric interface, or exclude the wrong ones with a caret: NCCL_SOCKET_IFNAME=^docker,lo,veth.
Connection refused during setup
Environment / runtime
NCCL WARN Call to connect returned Connection refused
A rank tried to reach a peer and nothing was listening. Either the peer had already exited, or the address it advertised is not reachable from the caller.
First action: Check whether the peer rank is alive before debugging the network: `grep -l Traceback rank-*.log` finds the one that died. A rank that crashed during init produces exactly this on every other rank.
An RDMA operation failed on the fabric. Error 12 with a vendor error is a transport-level fault, a link that dropped, a port that went down, or congestion severe enough to time out the queue pair.
First action: Check the port on both ends: ibstat and the error counters. Then test the raw path with ib_write_bw before blaming the application.
A rank asked for a GPU index that does not exist in its view. Usually CUDA_VISIBLE_DEVICES and the local rank disagreeing, so rank 3 asks for device 3 when it can only see one device.
First action: Print the mapping per rank: local rank, CUDA_VISIBLE_DEVICES and torch.cuda.device_count(). The launcher and the container device list are the two places this breaks.
Duplicate GPU across ranks
Configuration
NCCL WARN Duplicate GPU detected
Two ranks bound to the same physical GPU. NCCL refuses because a collective would deadlock against itself. Almost always a launcher assigning local ranks wrongly.
First action: Print the binding on every rank: LOCAL_RANK from the environment next to torch.cuda.current_device(). They must be one-to-one. Under torchrun the index is LOCAL_RANK; under Slurm it is usually SLURM_LOCALID.
The calling code broke an NCCL rule: mismatched tensor shapes or dtypes across ranks, collectives issued in a different order on different ranks, or a communicator used after being destroyed.
First action: Set TORCH_DISTRIBUTED_DEBUG=DETAIL, which validates that every rank issues the same collectives in the same order with the same shapes. A data-dependent branch in the forward pass is the usual culprit.
NCCL WARN Message truncated : received N bytes instead of M
Two ranks disagreed about the size of a collective. One sent more than the other expected, which means the ranks are running different shapes, and therefore probably different code or config.
First action: Run with TORCH_DISTRIBUTED_DEBUG=DETAIL and log `torch.distributed.get_rank()` alongside the tensor shape and dtype. Uneven batch splitting and a per-rank config difference are the two common causes.
Errors not listed here exist. Rather than guess at their meaning, check NVIDIA's NCCL troubleshooting guide, which is the authority for the strings above.
Environment variables worth knowing
Most of these are diagnostics rather than fixes. If one makes a failure disappear, it has told you where the fault is, not removed it.
Environment variables and what each one does.
Variable
What it does
NCCL_DEBUG=INFO
The single most useful setting. Prints transport selection, topology and ring construction. Start here for any NCCL problem.
NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH
Narrows the INFO firehose to setup, network and topology. Use with NCCL_DEBUG=INFO.
NCCL_SOCKET_IFNAME
Which interface to use for bootstrap and socket transport. A caret prefix excludes instead of includes: ^docker,lo.
NCCL_IB_DISABLE=1
Forces TCP instead of InfiniBand. A diagnostic, not a fix: if it makes the problem disappear, the fault is in the IB path.
NCCL_P2P_DISABLE=1
Disables direct GPU-to-GPU transfers. Another bisection tool; it will be slower.
NCCL_ALGO
Forces Ring or Tree. Useful when auto-selection picks an algorithm that stalls on an irregular rank count.
NCCL_IB_HCA
Binds ranks to a specific HCA. Needed on multi-NIC nodes so a rank uses the NIC on its own NUMA node.
TORCH_NCCL_TRACE_BUFFER_SIZE
Enables the PyTorch flight recorder. On a timeout it dumps which collective each rank was in, which is how you find the missing rank.
Frequently asked questions
Why does NCCL report the error when the problem is somewhere else?
Because NCCL is where the job synchronises. A rank that dies from an OOM, an Xid or a segfault stops participating in collectives, and every other rank discovers this when its next collective times out. The surviving ranks report; the dead one does not. That is why the loudest error is almost never the cause.
How do I find which rank actually failed?
Enable the PyTorch flight recorder with TORCH_NCCL_TRACE_BUFFER_SIZE=2000. On a timeout it records which collective each rank was executing, so the rank that never entered the collective is visible directly. Without it, scan every rank log by timestamp and take the earliest distinct error, not the most frequent one.
Does NCCL_IB_DISABLE=1 fix my problem?
It is a bisection tool, not a fix. If disabling InfiniBand makes the failure go away, you have proven the fault is in the IB path and you should fix it there. Leaving it set ships a job that runs over TCP, which on a multi-node training run is a very large and permanent throughput loss.
Why does it work on one node and hang on two?
Single-node collectives go over NVLink or PCIe and never touch the network. The moment a second node joins, bootstrap, interface selection, MTU, firewall rules and RDMA all enter the picture. A failure that appears only at two nodes is a network or transport problem, not a model problem.
What is the difference between a hang and a timeout?
They are the same event at different stages. The collective stalls, and after the timeout window the watchdog aborts the process and prints. Raising the timeout does not fix anything; it only delays the report and makes the wasted GPU time longer.
Knowing the error is not knowing the cause
Which rank failed first, whether the checkpoint is safe to resume from, and whether this is your code or the hardware. Paste the log and Denpex answers all three, free and without an account.