InfiniBand vs RoCEv2 GPU Cluster Errors & NCCL Triage
Connecting thousands of GPUs requires ultra-low latency fabrics via InfiniBand HDR/NDR or RoCEv2 (RDMA over Converged Ethernet). Fabric misconfigurations trigger subtle NCCL lockups, mlx5 driver errors, and Priority Flow Control (PFC) storms. This guide compares network troubleshooting workflows.
Comparison Matrix
| Dimension | InfiniBand (HDR / NDR) | RoCEv2 (RDMA over Converged Ethernet) | Shared Network Triage |
|---|---|---|---|
| Fabric Architecture | Credit-based lossless flow control natively built into InfiniBand architecture. | Lossless Ethernet requiring Priority Flow Control (PFC) and Explicit Congestion Notification (ECN). | NCCL collective communication relying on point-to-point RC/DC transport. |
| Primary Error Signatures | NCCL WARN NET/IB : Got completion with error 12, opcode 0, vendor err 129. | PFC storm, packet drop surge, ECN marking rate spike, and MTU mismatch. | NCCL WARN Connect failed: Connection refused or network interface unreachable. |
| Configuration Pitfalls | Subnet manager (OpenSM) partition key mismatches and LID assignment drops. | Incorrect GID index selection, VLAN tagging errors, and jumbo frame MTU 9000 omission. | NCCL_IB_HCA or NCCL_IB_GID_INDEX environment variable misconfigurations. |
| Diagnostic Utilities | ibstat, ib_write_bw, ibnetdiscover, and mlxreg tool suite. | ethtool, tcpdump, dcb tool, and roce_stats monitoring counters. | NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,NET environment flags. |
| Mitigation Strategy | Restart OpenSM, check cable seating, and replace degraded HDR/NDR transceivers. | Tune switch buffer thresholds, enforce strict lossless QoS, and verify DSCP markings. | Isolate failing HCA ports and update Mellanox/NVIDIA OFED drivers. |
Signature failures and first triage commands
InfiniBand Vendor Error 129
NCCL WARN NET/IB : Got completion with error 12, opcode 0, vendor err 129, dest rank 5.ibstat && ibv_devinfo -vRoCEv2 GID Index Mismatch
NCCL WARN Failed to find valid GID index for RoCE interface; check NCCL_IB_GID_INDEX setting.show_gids | grep -i RoCEEthernet PFC Pause Frame Storm
ethtool -S eth0 | grep -i pfc_requests_rx (Pause frames incrementing rapidly without data throughput).ethtool -S eth0Engineering decision guide
Choose InfiniBand for Guaranteed Lossless Performance
InfiniBand HDR/NDR provides native hardware-level credit flow control, eliminating the need for complex Ethernet PFC tuning and guaranteeing consistent multi-node bandwidth.
Choose RoCEv2 for Standardized Enterprise Ethernet Integration
RoCEv2 allows running RDMA workloads over existing datacenter Ethernet switches, but requires meticulous switch-level QoS, PFC, and ECN configuration to prevent deadlocks.
Standardize Network Health Checks
Run pre-flight IB/RoCE bandwidth benchmarks (ib_write_bw) and NCCL test suites before every large training run to catch degraded links early.
Frequently asked questions
What does vendor err 129 mean in NCCL InfiniBand logs?
Vendor error 129 typically indicates a transport-level retry counter exhaustion or link layer failure on the Mellanox/NVIDIA HCA, often caused by signal integrity degradation or firmware mismatch.
Why is RoCEv2 so sensitive to switch configuration?
Unlike InfiniBand, Ethernet is inherently lossy. If Priority Flow Control (PFC) is misconfigured on switches, packet drops trigger TCP-like retransmissions that destroy distributed training throughput.
How do I find the correct GID index for RoCEv2?
Run show_gids to list available Global Identifier entries on your network interface and set NCCL_IB_GID_INDEX to the index corresponding to RoCE v2 (typically index 1 or 3 depending on IPv4/IPv6).