Communication error references

Browse 90 published references for communication failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• ACS Disabled. GPU Direct RDMA Failure (Vendor Err 81)

  When Access Control Services (ACS) is disabled in the BIOS, GPU Direct RDMA (GDR) fails with RDMA completion error 4 and vendor error 81. NCCL performance degrades severely or training crashes. Denpex detects the GDR failure from NCCL logs and recommends the BIOS or NCCL configuration fix.

• All-Reduce Deadlock

  All-reduce deadlocks occur when DDP/FSDP all-reduce operations are mismatched across ranks or with batch norm.

• Asymmetric Collective Calls leading to NCCL Watchdog Timeout

  Different ranks in the distributed process group execute a different sequence of collective operations (e.g., one rank hits an `if` condition and calls `all_reduce`, while another skips it). Since NCCL relies on matching collective calls across all ranks synchronously, the ranks that called the collective wait forever for the others.

• AWS EFA Driver Issue

  AWS EFA (Elastic Fabric Adapter) driver issues cause NCCL to fall back to slower TCP/IP for inter-node communication.

• broadcast_coalesced Fails with Cuda failure 1 'invalid argument'

  DataParallel model replication fails at the SharedInit step with a CUDA invalid-argument error, often tied to legacy nn.DataParallel inside complex launchers.

• DDP Port Conflict

  DDP port conflicts prevent distributed training from starting when the master port is already in use.

• DDP Setup Error

  DDP setup errors occur when distributed data parallel training is not properly configured, preventing multi-GPU training.

• DNS Resolution Failure Cascading into Rendezvous and NCCL Bootstrap Errors

  When cluster DNS degrades, every layer above it fails with ITS own vocabulary: gaierror in python, client-socket failures in torch.distributed, bootstrap failures in NCCL. Teams debug the fabric while the resolver is the fault.

• FSDP Ignores User-Defined NCCL Timeout with Device Mesh

  When FSDP uses `device_mesh` or certain sharding strategies like `HYBRID_SHARD`, it creates its own internal sub-process groups for inter-node and intra-node communication. These internally created process groups do not inherit the custom `timeout` specified by the user in `dist.init_process_group()`. Instead, they silently default to the standard PyTorch 30-minute timeout.

• FSDP ncclSystemError on InfiniBand (works on TCP)

  FSDP all-gather fails with ncclSystemError on one node during collective init over InfiniBand, while a TCP fallback succeeds. Pointing to a per-node fabric or plugin issue.

• gdrcopy gdr_pin_buffer failed GPU virtual address is not mapped to BAR1 aperture

  GDRCopy could not pin the GPU virtual address into the CPU-visible BAR1 aperture. The allocation is not eligible for peer-direct mapping, BAR1 space is exhausted, or the kernel module and driver disagree. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent gdr-fabric failures.

• Gloo Backend Issue

  Gloo backend issues occur when using Gloo instead of NCCL for distributed training, especially on CPU or limited GPU setups.

• Gloo Connection refused to host 29500 during initial rendezvous

  Gloo TCP store connection failed (connection refused or DNS resolution failed), ranks cannot reach the rendezvous point. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent gdr-fabric failures.

• Horovod Setup Error

  Horovod setup errors occur when MPI or NCCL integration is misconfigured, preventing elastic or multi-GPU training.

• ibv_rc_pingpong: Failed to modify QP to RTR errno 110 Connection timed out

  An RDMA queue pair could not transition to Ready-To-Receive, so the connection never established. On RoCE this is nearly always a GID index or MTU mismatch between the two endpoints; on InfiniBand it points at the subnet manager not having configured the path. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent infiniband failures.

• Mixture-of-experts dispatch and combine collectives time out on large NVLink domains

  Expert-parallel routing sends every token to the ranks holding its chosen experts and gathers the results back, an all-to-all exchange whose volume depends on what the router chose. On large NVLink domains this pair of collectives can stop completing, and the job stalls inside it with no rank reporting a fault.

• mlx5_core mlx5_pfc_stall_detect PFC storm detected on priority fabric deadlock

  Priority Flow Control pause frames saturated a link. PFC exists to make RoCE lossless, but a congested or misconfigured fabric can propagate pause backward through switches, a "congestion tree" that stalls traffic far from the original hotspot. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent gdr-fabric failures.

• NCCL 'Could not find NET with id 0' (NIC Fusion)

  Intermittent NCCL internal errors during init on partial-node allocations, caused by the NIC-fusion feature remapping NET IDs so the hardcoded fallback NET/0 no longer exists.

• NCCL AllReduce Hang

  NCCL allreduce hangs stall training when collective operations can't complete. Denpex identifies the stuck rank or rank pair.

• NCCL Asynchronous Error Handling

  NCCL asynchronous error handling issues cause silent failures or deadlocks when collective errors are not properly handled.

• NCCL Bad RDMA Performance

  NCCL RDMA performance degrades when GPUDirect RDMA is unavailable or misconfigured, causing slow multi-node training.

• NCCL Binding to Virtual/Docker Interface

  When multiple network interfaces are present (e.g., physical eth0, physical high-speed mlx5_0, virtual docker0, loopback), NCCL uses a heuristic to automatically select the network interface for communication. In containerized environments, it frequently selects a virtual interface like `docker0` or `flannel.1` which cannot route traffic across physical nodes.

• NCCL Bucket Size Mismatch

  NCCL bucket size mismatches in DDP cause inefficient gradient reduction, hurting performance.

• NCCL BUFFSIZE Oversized, Socket Transport Stall

  Setting NCCL_BUFFSIZE too large causes NCCL Socket transport to stall or hang during communication. This is a configuration error where users increase the buffer size hoping for better performance but exceed the socket buffer limits. Denpex detects the oversized buffer from NCCL debug logs.

• NCCL Collective Operation Timeout

  A NCCL collective operation failed to complete within the watchdog timeout period because one or more ranks stopped participating. This is the most common distributed training failure and is almost always a downstream symptom of a different root cause (GPU crash, OOM, network loss, or host failure).

• NCCL Communication Timeout

  NCCL communication timeouts abort training when collective operations take too long to complete.

• NCCL CUDA Failure

  NCCL CUDA failures occur when the underlying CUDA runtime has issues, causing all NCCL operations to fail.

• NCCL Error 2: Internal Error

  NCCL error 2 is an internal assertion failure in NCCL itself, often from corrupted state or hardware issues. Denpex correlates with hardware diagnostics.

• NCCL GPUDirect over RoCE

  NCCL GPUDirect over RoCE (RDMA over Converged Ethernet) requires proper configuration for high performance.

• NCCL Hang Detection

  NCCL hangs can be hard to detect because they don't produce errors until the watchdog timeout. Proper monitoring helps catch them early.

• NCCL Hang on AllGather due to Asymmetric Data Loading

  ZeRO-3 requires all ranks to participate in collective communication (`AllGather`) to reconstruct parameter layers during the forward and backward passes. If `drop_last=False` is set in the PyTorch DataLoader, the final batch of an epoch may be uneven. A rank that receives a smaller batch (or finishes its data early) exits the training loop, while other ranks hit the `AllGather` barrier and wait indefinitely.

• NCCL Hangs with Exactly Three InfiniBand NICs

  NCCL's NIC selection/path-finding can deadlock when a node has an odd number of HCAs (specifically three), while 1, 2, or 4+ work fine.

• NCCL IB HCA Mismatch

  NCCL InfiniBand HCA (Host Channel Adapter) configuration mismatches cause slow or failed inter-node communication.

• NCCL Initialization Error

  NCCL initialization errors prevent distributed training from starting due to network or configuration issues.

• NCCL Initialization Timeout

  NCCL initialization times out when nodes cannot establish the communication channel within the allowed time.

• NCCL IPv6 Issue

  NCCL IPv6 issues occur when NCCL tries to use IPv6 but the network only supports IPv4, or vice versa.

• NCCL Mismatched Collective. Different Ranks Executing Different Operations

  NCCL mismatched collective errors occur when different ranks call different collective operations or call the same collective with different parameters. This is a programming error, not a hardware fault. Denpex identifies which rank diverged and what operation it attempted.

• NCCL MNNVL Init Segfault at Large World Size (Stack Overrun)

  On GB200 NVL72, NCCL's recursive MNNVL topology search overruns the stack at world_size >= 44, especially with ulimit -s unlimited; fixed in NCCL 2.28.

• NCCL multi-NIC packet reordering detected RoCE packets arrived out of sequence pipeline stalled

  Traffic spread across multiple NICs arrived out of order. RoCE is highly sensitive to reordering, go-back-N retransmission means a reordered flow collapses throughput rather than degrading gracefully, and can stall the pipeline entirely. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent nccl-tuning failures.

• NCCL Network Topology Lookup Failure (NIC Fusion)

  NCCL cannot find a valid path for the network pattern in the discovered PCIe topology. This is typically caused by NIC fusion (NIC split) remapping NIC device IDs during partial-node InfiniBand allocations.

• NCCL NVLink Routing Loop Fallback Hang

  A single faulty NVLink lane causes NCCL to detect topology degradation. NCCL attempts to dynamically route traffic around the bad link using InfiniBand (IB) or PCIe as a fallback. Due to a bug in older NCCL versions or complex topology configs, this fallback creates an infinite routing loop or buffer deadlock in the collective communication ring.

• NCCL NVLS CUDA failure 1: invalid argument

  This NCCL warning occurs while the NVLink SHARP transport is creating or registering multicast resources. The line identifies the failing NVLS path, but it does not prove whether the owner is application input, NCCL, CUDA, Fabric Manager, or the NVSwitch domain.

• NCCL NVLS Memory Corruption with Dual-Port NICs

  NVLS did not support dual-port NIC transmission, producing duplicate head-rank entries in the proxy loop and heap corruption; fixed in NCCL 2.19.4.

• NCCL P2P Bandwidth Issue

  NCCL point-to-point bandwidth issues cause slow intra-node GPU communication when P2P is disabled or limited.

• NCCL P2P Communication Issue

  NCCL peer-to-peer (P2P) communication issues cause pipeline parallelism and tensor parallelism to fail or be slow.

• NCCL P2P Disabled

  NCCL P2P (peer-to-peer) communication can be disabled, forcing fallback to slower transport.

• NCCL P2P Fails on RTX 5090 / Blackwell (SM120)

  NCCL P2P topology detection lacks SM120 (Blackwell) support, using a wrong shared-memory maximum so peer-to-peer connections fail on RTX 5090.

• NCCL Proxy Connect Failed (IPv6 Interference)

  NCCL's proxy thread tries to connect over IPv6 while the peer only answers on IPv4, producing 'Proxy Connect failed' on dual-stack hosts.

• NCCL Random Segfault from Out-of-Order NIC Names

  Inconsistent NIC enumeration (e.g. mlx5_3 before mlx5_0) on one node breaks NCCL topology detection and causes random multi-node segfaults; fixed in NCCL 2.23.4.

• NCCL Rank Fail

  NCCL rank failures occur when one or more ranks fail to initialize or join the distributed group.

• NCCL Rank Stuck / Straggler

  NCCL rank stuck errors occur when one rank cannot keep up with the collective operation, slowing down all ranks.

• NCCL RAS Query Crashes Job (Memory Corruption, 2.27.3)

  Querying the NCCL RAS subsystem during training on NCCL 2.27.3 returns corrupted communicator data and can crash the job via TCPStore connection failures.

• NCCL RAS Race Segfault During Initialization

  A race in the NCCL RAS subsystem segfaults during init when a RAS command runs before peer info is populated; fixed in NCCL 2.27.3.

• NCCL ring buffer allocation failed NCCL_BUFFSIZE exceeds available contiguous pinned host memory

  NCCL could not allocate the contiguous pinned host buffer requested by NCCL_BUFFSIZE. The configured buffer and communicator count exceeded available lockable memory or the process memlock limit. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent nccl-tuning failures.

• NCCL RoCE GID Read Failed (Invalid argument)

  NCCL fails to read the RoCE GID on containerized/macvlan setups because GID iteration starts at index 0 where leading entries are all-zero, returning EINVAL.

• NCCL RoCEv2 GID Index Mismatch

  RoCE (RDMA over Converged Ethernet) NICs support multiple protocols simultaneously (IPv4, IPv6, RoCEv1, RoCEv2), mapped to different GID (Global Identifier) indices. By default, NCCL might attempt to use GID Index 0 (often RoCEv1 or IPv6). If the network switches are configured strictly for IPv4 RoCEv2, the packets generated by GID 0 are deemed unroutable and dropped silently.

• NCCL socketStartConnect: Software caused connection abort

  A stale socket file descriptor is reused on retry after a failed connect, triggering ECONNABORTED during NCCL communicator init; fixed in NCCL 2.24.

• NCCL Timeout Cascade from Single Rank Failure

  A single rank failure cascades into mass NCCL timeouts across most or all ranks as they wait at a collective barrier. The cascade signature is one rank failing first, followed by a wave of timeouts on all other ranks.

• NCCL Timeout During Synchronous Checkpoint Save

  Synchronous checkpointing blocks the training loop. On slow shared storage (e.g., NFS), Rank 0 takes a long time to write the file. Meanwhile, other ranks reach the next distributed collective (e.g., all-reduce) and wait. If the wait exceeds the NCCL timeout (default 30 mins), the watchdog kills the process.

• NCCL Timeout Triggered by Hidden Illegal Memory Access

  A completely different, localized error (like an out-of-bounds tensor access, device-side assert, or OOM) occurred on a *single* GPU in the distributed cluster. Because that GPU's CUDA context crashes, it silently stops participating in NCCL collective communications. The other healthy GPUs block indefinitely waiting for it, eventually triggering an NCCL timeout.

• NCCL Topology Detection Regression (v2.18.3+)

  NCCL v2.18.3 introduced a topology detection regression that misidentifies GPU-NIC affinity on some systems, causing NCCL to use a suboptimal communication path. Denpex detects the regression from NCCL version and topology logs.

• NCCL Topology XML Missing NIC Bus ID

  NCCL topology detection fails when the NIC's PCI bus ID is missing from the system topology XML. This causes NCCL to fall back to a suboptimal transport or fail to use GPU Direct RDMA. Denpex detects the missing bus ID and recommends the configuration override.

• NCCL Tree algorithm deadlocked non-power-of-two rank count NCCL_ALGO Ring

  The NCCL tree algorithm stalled on this topology. Tree is chosen automatically for some rank counts and message sizes, and an irregular or asymmetric rank layout can leave it waiting on a peer that never sends. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent nccl-tuning failures.

• NCCL Vendor Err 129 (RDMA Timeout)

  Vendor Error 129 represents an InfiniBand transport retry counter exceeded error. In a RoCEv2 environment, this typically happens because the network is lossy and packets are being dropped by the switch due to congestion. RoCEv2 requires a lossless fabric (via Priority Flow Control / PFC) or properly configured congestion control (DCQCN). When packets drop, the NIC retries sending until the retry limit is hit, yielding error 129.

• NCCL Version Conflict

  NCCL version conflicts prevent proper collective operations when nodes have different versions installed.

• NCCL Version Mismatch

  NCCL version mismatches between nodes cause collective operations to fail or hang in distributed training.

• NCCL WARN Bootstrap: Condition [state == ncclSuccess] failed: Connection reset by peer

  An NCCL bootstrap TCP connection was accepted and then reset by a peer before communicator setup completed. The resetting rank may have exited, selected a different interface, or lost network reachability during initialization. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent nccl failures.

• NCCL WARN Connection failed : Network is unreachable NCCL_SOCKET_IFNAME

  NCCL tried to reach a peer on an interface that cannot route to it. Almost always NCCL_SOCKET_IFNAME selecting the wrong NIC, a docker0/lo/management interface instead of the fabric, so ranks advertise addresses their peers cannot reach. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent nccl failures.

• NCCL WARN Direct RDMA write between GPU NUMA and HCA NUMA failed UPI interconnect

  GPUDirect RDMA was attempted between a GPU and a NIC on different NUMA nodes. The transfer has to cross the CPU interconnect (UPI/Infinity Fabric), which is far slower than a local path and on some platforms is not supported for peer-to-peer at all. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent gdr-fabric failures.

• NCCL WARN Failed to initialize transport NET IB Resource temporarily unavailable

  NCCL could not bring up a transport (IB/EFA/NET) between two ranks. "Resource temporarily unavailable" here usually means the RDMA device ran out of queue pairs or memory registrations, or the container lacks the device/ulimit needed to allocate them. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent nccl failures.

• NCCL WARN Ring via NET/Socket Call to connect returned Connection refused

  An NCCL ring peer attempted a TCP connection before the expected listener was reachable. The destination rank may have crashed, advertised the wrong interface, or been blocked by host or network policy. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent nccl failures.

• NCCL Watchdog Timeout due to Incorrect Network Interface

  Nodes have multiple network interfaces (e.g., a high-speed InfiniBand/RoCE interface, a standard ethernet interface, and a Docker bridge). NCCL heuristically picks an interface, often picking a local virtual interface (like docker0) that cannot route traffic to other nodes. Processes attempt to form the NCCL ring but packets are dropped, leading to a hang.

• NCCL Wrong Rank Configuration

  Wrong rank configuration in NCCL causes collectives to fail or produce incorrect results when rank assignments don't match expectations.

• ncclCommSplit Segfault with Non-Blocking Init

  A dangling group-job pointer causes a segfault during ncclCommSplit when using threads with non-blocking communicator init; fixed in NCCL 2.26.2.

• ncclUnhandledCudaError: Cuda failure 999 'unknown error'

  A generic CUDA 'unknown error' (999) surfaces during NCCL communicator initialization, usually pointing to a GPU in a bad state or a driver/runtime mismatch rather than an NCCL bug.

• nvidia-peermem kernel rejected peer-direct DMA mapping BAR1 aperture exhausted

  nvidia-peermem could not create another peer-direct mapping because the GPU BAR1 aperture has no usable space for it. Stale peer mappings, an undersized BAR1 window, or an unsupported platform layout can produce the same rejection. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent infiniband failures.

• NVLink SHARP (NVLS) multicast initialization failure

  NVLS initialization failures occur while NCCL is preparing NVSwitch multicast resources for collective offload. Diagnose support, allocation, software state, and fabric health separately instead of treating every NVLS message as the same defect.

• NVLS Multicast Slot Exhaustion (NVSwitch)

  NCCL exhausts the NVSwitch's fixed pool of NVLink SHARP (NVLS) multicast slots, then treats the binding failure as fatal instead of falling back to a non-NVLS transport.

• OFI Memlock Exhaustion. RDMA Memory Registration Failure

  NCCL's OFI (OpenFabrics Interface) transport fails to register memory for RDMA when the OS memlock limit is too low. Training crashes at NCCL init with 'Unable to register memory RC:12'. Denpex detects the insufficient memlock limit from NCCL logs and recommends the system-level fix.

• OpenSM SM sweep failed: Link state active timeout on Port GUID

  InfiniBand Subnet Manager Flap - The fabric routing changed during the run, dropping active connections. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent infiniband failures.

• Pipeline Parallel Bubble

  Pipeline parallelism has idle time (bubble) at start and end of each pipeline, reducing efficiency for small models or small micro-batches.

• RDMA Configuration Issue

  RDMA configuration issues prevent high-bandwidth, low-latency inter-node communication in GPU clusters.

• RoCE MTU Mismatch. Completion Error 12 / Vendor Err 129

  RoCE MTU mismatches between the NIC and switch cause NCCL RDMA queue-pair errors (completion error 12, vendor error 129). Training hangs or crashes during NCCL init or the first collective. Denpex detects the MTU mismatch from NCCL debug logs and identifies the misconfigured endpoint.

• RoCE QP Timeout During Distributed Checkpoint Save

  NCCL queue-pair creation times out specifically during distributed checkpoint save at large scale on RoCE, while normal training collectives succeed.

• TCP Port Exhaustion

  TCP port exhaustion occurs in distributed training when many connections exhaust ephemeral port range.

• TorchElastic Error

  TorchElastic errors occur when training jobs need to be elastic (resize dynamically) but configuration is wrong.

• TorchElastic Rendezvous Timeout on Large Clusters

  The default `torchrun` rendezvous timeout is 600 seconds. In large scale clusters (e.g., hundreds of nodes on AWS/GCP), container startup times, image pulling, and internal network resolution can easily exceed 10 minutes, causing early nodes to time out and crash before late nodes finish booting up. Additionally, short hostnames might fail to resolve if DNS is misconfigured, causing silent connection drops.

• TP Comm Overlap Topology Conflict Deadlock

  The `--tp-comm-overlap` feature schedules communication kernels concurrently with compute kernels using separate CUDA streams. On systems lacking dense NVLink (where P2P must go through PCIe), the PCIe bus becomes saturated. This saturation delays the communication stream to the point where stream synchronization primitives (CUDA events) trigger out of order, or NCCL drops into a fallback protocol that deadlocks when interleaved with heavy compute kernels.

• Variable Sequence Length Shape Exchange Deadlock

  When `--variable-seq-lengths` and `batch_p2p_comm=True` are enabled with Pipeline Parallelism (PP >= 4), the `_communicate_shapes()` function enforces a strict, fixed point-to-point operation ordering: [send_prev, recv_prev, send_next, recv_next]. During the transition phases of the pipeline schedule, this strict ordering causes circular waits (Rank A waiting on Rank B, while Rank B waits on Rank A for a different operation), leading to a distributed deadlock.

• vLLM workers time out reading the shared-memory broadcast ring and the engine dies

  vLLM distributes each step to its workers through a shared-memory ring buffer. When a worker fails to publish within the read deadline, the reader raises a bare TimeoutError from acquire_read, the execute_model call that was waiting on it fails, and the engine tears down, so a stalled worker is reported as a timeout in the transport rather than as a problem in the worker.