What is the purpose of NCCL_IB_GID_INDEX?

NCCL_IB_GID_INDEX tells the NVIDIA Collective Communications Library which GID table entry to use on RoCE interfaces. Using RoCE v2 with IPv4-mapped GIDs is required for routed multi-subnet GPU clusters.

How is NCCL_IB_TC calculated from DSCP?

NCCL_IB_TC is typically calculated by multiplying the DSCP value by 4 (DSCP << 2) to align with the Traffic Class bits in the IPv4 header TOS field.

Why do RoCEv2 GPU clusters experience hangs during distributed training?

Lossless Ethernet fabrics require Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) correctly configured across all top-of-rack switches to prevent buffer starvation and deadlocks.

What is the recommended interface MTU for RoCEv2 traffic?

An MTU of 9000 bytes (Jumbo Frames) is strongly recommended for RoCEv2 traffic to reduce packet fragmentation and overhead during high-throughput collective operations.

Denpex Infrastructure Tooling

RoCEv2 and InfiniBand NCCL GID Index Calculator

Q: Why do RoCEv2 GPU clusters experience hangs during distributed training?

Lossless Ethernet fabrics require Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) correctly configured across all top-of-rack switches to prevent buffer starvation and deadlocks.

Q: What is the recommended interface MTU for RoCEv2 traffic?

An MTU of 9000 bytes (Jumbo Frames) is strongly recommended for RoCEv2 traffic to reduce packet fragmentation and overhead during high-throughput collective operations.

Q: How does Denpex diagnose NCCL and GPU communication failures?

Denpex automatically diagnoses distributed training failures across PyTorch, FSDP, DeepSpeed, CUDA, and NCCL on GPU clusters, pinpointing network timeouts and configuration mismatches instantly.

Parse host GID tables, compute optimal NCCL environment variables, configure lossless Ethernet Quality of Service parameters, and prevent multi-node congestion deadlocks across GPU clusters.

Explore Denpex Platform

Denpex Enterprise Team Evaluation

Automate distributed training failure diagnosis across PyTorch, FSDP, DeepSpeed, CUDA, and NCCL on GPU clusters with our 30-day free trial.

Start 30-Day Free Trial

Raw GID Table Output (show_gids or ibv_devinfo -v)

Parsed 4 GID entries successfully. Select a specific GID entry below or rely on automated RoCEv2 IPv4 selection.

NCCL and Fabric Tuning Parameters

DSCP Value (Default: 26)

AF31 (26)EF (46)CS6 (48)

NCCL Net GDR Level

RoCE MTU (Bytes)

Interface MTU

HCA Device Override

Parsed GID Table and Index Selection

Idx	Device	Version	IP Address
0	mlx5_0:1	RoCE v1	Link-Local
1	mlx5_0:1	RoCE v1	10.0.1.5
2	mlx5_0:1	RoCE v2	Link-Local
3	mlx5_0:1	RoCE v2	10.0.1.5

Recommended GID index: 3 (RoCE v2 with IPv4 mapping).

Generated NCCL Environment Variables

export NCCL_IB_GID_INDEX=3
export NCCL_IB_TC=104
export NCCL_IB_SL=3
export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_HCA=mlx5_0:1
export NCCL_IB_QPS_PER_CONNECTION=4

Calculated Traffic Class (TC)DSCP 26 × 4 = 104

Service Level (SL)3

Lossless Ethernet Switch Configuration Recipes

Apply matching DSCP to Traffic Class maps on your top-of-rack switches.

Vendor OS: aristaTarget DSCP: 26

class-map type qos match-any RDMA
  match dscp 26
policy-map type qos QOS-POLICY
  class RDMA
    set traffic-class 3

RoCEv2 Deadlock and Hang Prevention Checklist

Ensure PFC (Priority Flow Control) is enabled on the switch for the specified Priority/Traffic Class.

Verify ECN (Explicit Congestion Notification) is active on the switch for traffic mapped to DSCP 26.

Check that the firewall (e.g. iptables/ufw) is not dropping UDP traffic on RoCE v2 ports.

Run ib_write_bw or rping before running NCCL to validate native RDMA connectivity.