NCCL Mismatched Collective. Different Ranks Executing Different Operations

NCCL mismatched collective errors occur when different ranks call different collective operations or call the same collective with different parameters. This is a programming error, not a hardware fault. Denpex identifies which rank diverged and what operation it attempted.

NCCL Mismatched Collective. Different Ranks Executing Different Operations is a Communication failure seen during ML training runs. NCCL mismatched collective errors occur when different ranks call different collective operations or call the same collective with different parameters. This is a programming error, not a hardware fault. Denpex identifies which rank diverged and what operation it attempted. Common tags: Nccl, Collective, Mismatch, Ddp.

A conditional branch (if/else) causes different ranks to execute different collective operations at the same point in the training loop. One rank skips a collective (e.g., skips backward pass for a batch with no loss) while other ranks execute it. Dynamic model architectures (MoE routing, early exit) produce different collective patterns on different ranks. Mismatched data loading: one rank processes a different number of batches, causing it to be one collective ahead or behind. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training crashes with 'Mismatched collective' error from NCCL
• One rank calls all-reduce while another calls all-gather at the same collective step
• The error is deterministic and reproducible with the same code path

• DDP/FSDP training with conditional computation paths
• Models with dynamic computation graphs that vary across ranks
• MoE models where different experts are activated on different ranks
• Training loops with early stopping or gradient skipping that isn't synchronized

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: NCCL error: Mismatched collective detected at rank X
• ✓Verified signal present: NCCL_DEBUG=INFO shows different collective types on different ranks at the same step
• ✓Verified signal present: Training hangs before the mismatch error if ranks are waiting at different barriers
• ✓Verified signal present: The error occurs after a conditional branch that differs across ranks
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.

• A conditional branch (if/else) causes different ranks to execute different collective operations at the same point in the training loop
• One rank skips a collective (e.g., skips backward pass for a batch with no loss) while other ranks execute it
• Dynamic model architectures (MoE routing, early exit) produce different collective patterns on different ranks
• Mismatched data loading: one rank processes a different number of batches, causing it to be one collective ahead or behind