RoCE MTU Mismatch. Completion Error 12 / Vendor Err 129

RoCE MTU mismatches between the NIC and switch cause NCCL RDMA queue-pair errors (completion error 12, vendor error 129). Training hangs or crashes during NCCL init or the first collective. Denpex detects the MTU mismatch from NCCL debug logs and identifies the misconfigured endpoint.

RoCE MTU Mismatch. Completion Error 12 / Vendor Err 129 is a Communication failure seen during ML training runs. RoCE MTU mismatches between the NIC and switch cause NCCL RDMA queue-pair errors (completion error 12, vendor error 129). Training hangs or crashes during NCCL init or the first collective. Denpex detects the MTU mismatch from NCCL debug logs and identifies the misconfigured endpoint. Common tags: Roce, Mtu, Infiniband, Rdma.

The NIC MTU (active_mtu from ibstat) does not match the switch port MTU, causing RDMA packet fragmentation or drops that manifest as completion error 12. Vendor error 129 is Mellanox-specific: the NIC's firmware rejected the packet because the MTU exceeded the port's configured maximum. After a switch firmware upgrade, the default MTU may change from 2048 to 4096 without updating the NIC configuration. Docker containers may inherit the host MTU but the RoCE device has a different MTU configured. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• NCCL init succeeds but the first all-reduce or all-gather fails with completion error 12
• Training hangs at NCCL communication with ibv completion errors in dmesg
• Single-node training works but multi-node RoCE fails immediately

• Multi-node training over RoCEv2 (Mellanox ConnectX-5/6/7)
• AWS EFA-based clusters with custom MTU settings
• Bare-metal clusters with mixed switch firmware
• Environments where switch MTU was changed without updating NIC MTU

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: NCCL WARN Got completion with error 12, vendor err 129
• ✓Verified signal present: ibv_post_send fails on specific node pairs but not others
• ✓Verified signal present: NCCL_DEBUG=INFO shows IB transport errors during channel setup
• ✓Verified signal present: ibstat shows active_mtu 2048 but switch port is configured for 4096 or vice versa
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.

• The NIC MTU (active_mtu from ibstat) does not match the switch port MTU, causing RDMA packet fragmentation or drops that manifest as completion error 12
• Vendor error 129 is Mellanox-specific: the NIC's firmware rejected the packet because the MTU exceeded the port's configured maximum
• After a switch firmware upgrade, the default MTU may change from 2048 to 4096 without updating the NIC configuration
• Docker containers may inherit the host MTU but the RoCE device has a different MTU configured