Mixture-of-experts dispatch and combine collectives time out on large NVLink domains

Expert-parallel routing sends every token to the ranks holding its chosen experts and gathers the results back, an all-to-all exchange whose volume depends on what the router chose. On large NVLink domains this pair of collectives can stop completing, and the job stalls inside it with no rank reporting a fault.

Symptom

File ".../bench/ep_harness.py", line 248, in sample

Root cause

It frequently does not present as a timeout at all, and that is the hardest part of recognising it. When a collective is abandoned the communicator is torn down while its kernels are still resident on the device. The next call that inspects device state, commonly torch.

Recommended fix

Establish whether the stall is in dispatch or in combine, since they are separate exchanges with different volumes and narrowing to one halves the search.

How Denpex helps

Denpex matches Mixture-of-experts dispatch and combine collectives time out on large NVLink domains across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

A stall in the all-to-all exchanges that implement mixture-of-experts routing, in which a dispatch or combine receive never completes on a large NVLink domain and the job halts inside the collective without any rank raising a fault.

Expert parallelism turns communication into a function of the data. The router decides which ranks exchange how much, so no two steps look alike and the pattern is neither uniform nor symmetric. All-to-all already puts every rank in contact with every other, and a rack-scale domain multiplies those simultaneous contacts, so assumptions that were safe inside one node meet a regime they were never exercised in.

• The stall is inside a collective, so no rank has an error of its own to report
• It depends on the data, because routing decides the exchange volume, and so it appears irregularly
• It emerges at large domain sizes and cannot be reproduced on a smaller topology
• Both the routing and the transport look correct in isolation

• Mixture-of-experts models using expert parallelism, where routing drives an all-to-all
• Rack-scale NVLink domains, where the number of simultaneous peer exchanges is far larger than in a single node
• Low-latency dispatch and combine kernels tuned for such domains
• Any workload whose collective volume is decided by data rather than fixed by shape

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Treat an unspecified launch failure at a synchronisation point as a downstream report, not a cause. Check whether an expert-parallel exchange preceded it in the same iteration before investigating kernels.
• ✓Determine whether the timing-out operation is the dispatch or the combine exchange; they are separate collectives and only one of them will be at fault.
• ✓Run the same model and data at a smaller domain size. Completing there and stalling at full scale indicates the number of concurrent peer exchanges is the factor.
• ✓Substitute a general-purpose collective for the specialised low-latency path and re-run, which isolates the kernel from the routing.

• It frequently does not present as a timeout at all, and that is the hardest part of recognising it. When a collective is abandoned the communicator is torn down while its kernels are still resident on the device. The next call that inspects device state, commonly torch.cuda.synchronize at the end of a benchmark iteration, is the one that collects the wreckage, and it reports an unspecified launch failure. The frame in the traceback is therefore innocent, and the exchange that stalled has already been cleaned up by the time anything is printed.
• Expert routing makes communication data-dependent. Which ranks exchange how much is decided per step by the router, so the traffic pattern changes from step to step and is not the uniform, symmetric shape that collective implementations are usually tuned and tested against.
• All-to-all is the most demanding of those patterns because every rank talks to every other simultaneously. Scaling the domain multiplies the number of concurrent peer exchanges rather than adding to it, so resource limits and ordering assumptions that hold within a node can fail across a rack.
• The receive side is where it surfaces because that is the side that waits. A dispatch that was not sent, or was sent to a peer that had already moved on, leaves a receiver blocked with nothing to report except that its data did not arrive.

  File ".../bench/ep_harness.py", line 248, in sample
    torch.cuda.synchronize()
torch.AcceleratorError: CUDA error: unspecified launch failure
dispatch/combine receives time out
[rank17] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLTOALL) ran for 1800000 milliseconds before timing out

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Narrowing to dispatch or combine, and to a domain size, replaces a whole-system symptom with a specific exchange at a specific scale, the only form in which it can be investigated. Falling back to a general-purpose collective separates the specialised kernel from the routing, and capping expert load removes the extreme imbalances that make the exchange hardest.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.