DeepSpeed bf16 Gradient Norm Underflow

DeepSpeed's bf16 training can trigger 'assert all_groups_norm > 0' because bf16 gradient norms can underflow to zero. This is a numerical precision issue specific to bf16's limited range. Denpex detects the underflow from DeepSpeed error logs and recommends the fix.

DeepSpeed bf16 Gradient Norm Underflow is a Training Stability failure seen during ML training runs. DeepSpeed's bf16 training can trigger 'assert all_groups_norm > 0' because bf16 gradient norms can underflow to zero. This is a numerical precision issue specific to bf16's limited range. Denpex detects the underflow from DeepSpeed error logs and recommends the fix. Common tags: Deepspeed, Bf16, Gradient Norm, Underflow.

Bf16 has fewer mantissa bits (7) than fp16 (10), reducing the smallest representable positive number. When gradient norms are very small (common with large models, small LR, or gradient accumulation), they can underflow to zero in bf16. DeepSpeed's gradient norm assertion assumes non-zero norms, which is not guaranteed with bf16 precision. The gradient norm is computed in bf16 precision instead of being upcast to fp32 for the norm calculation. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training crashes with 'assert all_groups_norm > 0' when using bf16 precision with DeepSpeed
• The error occurs even though gradients are non-zero (they're just too small for bf16 to represent)
• Switching to fp16 resolves the issue but bf16 is preferred for training stability

• DeepSpeed ZeRO stage 2/3 with bf16 precision
• Training with small learning rates that produce very small gradients
• Models with many layers where gradients diminish through backpropagation
• DeepSpeed versions before 0.13.5

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: AssertionError: all_groups_norm > 0 in DeepSpeed gradient norm computation
• ✓Verified signal present: Training runs fine with fp16 but crashes with bf16
• ✓Verified signal present: The error occurs at the same step consistently
• ✓Verified signal present: Gradient norms reported as 0.0 in DeepSpeed logs even though the model is learning
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

DeepSpeed changes when parameters, gradients and optimizer state are created, partitioned, gathered and offloaded. The hub separates ZeRO, memory, checkpoint and pipeline failures by lifecycle phase.

• bf16 has fewer mantissa bits (7) than fp16 (10), reducing the smallest representable positive number
• When gradient norms are very small (common with large models, small LR, or gradient accumulation), they can underflow to zero in bf16
• DeepSpeed's gradient norm assertion assumes non-zero norms, which is not guaranteed with bf16 precision
• The gradient norm is computed in bf16 precision instead of being upcast to fp32 for the norm calculation