Mixture-of-experts dispatch and combine collectives time out on large NVLink domains

Expert-parallel routing sends every token to the ranks holding its chosen experts and gathers the results back, an all-to-all exchange whose volume depends on what the router chose. On large NVLink domains this pair of collectives can stop completing, and the job stalls inside it with no rank reporting a fault.

A stall in the all-to-all exchanges that implement mixture-of-experts routing, in which a dispatch or combine receive never completes on a large NVLink domain and the job halts inside the collective without any rank raising a fault.

Expert parallelism turns communication into a function of the data. The router decides which ranks exchange how much, so no two steps look alike and the pattern is neither uniform nor symmetric. All-to-all already puts every rank in contact with every other, and a rack-scale domain multiplies those simultaneous contacts, so assumptions that were safe inside one node meet a regime they were never exercised in.

• The stall is inside a collective, so no rank has an error of its own to report
• It depends on the data, because routing decides the exchange volume, and so it appears irregularly
• It emerges at large domain sizes and cannot be reproduced on a smaller topology
• Both the routing and the transport look correct in isolation

• Mixture-of-experts models using expert parallelism, where routing drives an all-to-all
• Rack-scale NVLink domains, where the number of simultaneous peer exchanges is far larger than in a single node
• Low-latency dispatch and combine kernels tuned for such domains
• Any workload whose collective volume is decided by data rather than fixed by shape

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Determine whether the timing-out operation is the dispatch or the combine exchange; they are separate collectives and only one of them will be at fault.
• ✓Run the same model and data at a smaller domain size. Completing there and stalling at full scale indicates the number of concurrent peer exchanges is the factor.
• ✓Substitute a general-purpose collective for the specialised low-latency path and re-run, which isolates the kernel from the routing.

dispatch/combine receives time out
[rank17] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLTOALL) ran for 1800000 milliseconds before timing out

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Narrowing to dispatch or combine, and to a domain size, replaces a whole-system symptom with a specific exchange at a specific scale — the only form in which it can be investigated. Falling back to a general-purpose collective separates the specialised kernel from the routing, and capping expert load removes the extreme imbalances that make the exchange hardest.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.