DeepSpeed NaN from overlap_comm + contiguous_gradients

Enabling both overlap_comm and contiguous_gradients in DeepSpeed ZeRO-3 causes gradient buffer reuse races that produce NaN losses. This is a known DeepSpeed bug where the communication overlap reads from a gradient buffer that is being rewritten by the contiguous gradient copy. Denpex detects the specific flag combination and recommends disabling one.

DeepSpeed NaN from overlap_comm + contiguous_gradients is a Training Stability failure seen during ML training runs. Enabling both overlap_comm and contiguous_gradients in DeepSpeed ZeRO-3 causes gradient buffer reuse races that produce NaN losses. This is a known DeepSpeed bug where the communication overlap reads from a gradient buffer that is being rewritten by the contiguous gradient copy. Denpex detects the specific flag combination and recommends disabling one. Common tags: Deepspeed, Nan, Overlap Comm, Contiguous Gradients.

When overlap_comm is True, DeepSpeed starts communication (all-reduce) for one gradient bucket while the next bucket is being computed. When contiguous_gradients is True, DeepSpeed copies gradients into a contiguous buffer for efficient communication. The race: the communication overlap reads from the contiguous buffer while it's being rewritten by the next bucket's gradient copy, producing corrupted (NaN) values. This is a buffer reuse race condition in DeepSpeed's gradient communication pipeline. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training produces NaN gradient norms when both overlap_comm and contiguous_gradients are True in DeepSpeed ZeRO-3
• Disabling either flag individually resolves the NaN
• The NaN appears non-deterministically, making it hard to reproduce

• DeepSpeed ZeRO stage 3 with both overlap_comm and contiguous_gradients enabled
• Training on multi-node setups where communication overlap is critical for performance
• DeepSpeed versions before v0.14.0 where the race condition is not fixed

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: grad_norm is nan reported by DeepSpeed
• ✓Verified signal present: Training loss becomes NaN after a variable number of steps
• ✓Verified signal present: The NaN disappears when setting overlap_comm: false or contiguous_gradients: false
• ✓Verified signal present: DeepSpeed logs show NaN in gradient norm calculation
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

DeepSpeed changes when parameters, gradients and optimizer state are created, partitioned, gathered and offloaded. The hub separates ZeRO, memory, checkpoint and pipeline failures by lifecycle phase.

• When overlap_comm is True, DeepSpeed starts communication (all-reduce) for one gradient bucket while the next bucket is being computed
• When contiguous_gradients is True, DeepSpeed copies gradients into a contiguous buffer for efficient communication
• The race: the communication overlap reads from the contiguous buffer while it's being rewritten by the next bucket's gradient copy, producing corrupted (NaN) values
• This is a buffer reuse race condition in DeepSpeed's gradient communication pipeline