Training Stability error references

Browse 85 published references for training stability failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• Adafactor Optimizer Issues

  Adafactor optimizer issues arise from incorrect epsilon, scaling factor, or relative step handling.

• Adam Epsilon Hyperparameter Issue

  Adam epsilon hyperparameter issues cause training instability or poor convergence.

• AdamW Epsilon Underflow in FP16

  The default epsilon value for Adam/AdamW is 1e-8. In float16, the smallest representable positive normal number is ~5.96e-8. Thus, 1e-8 underflows to 0.0. During the optimizer update, this leads to a division by zero if the gradient variance estimate is also close to zero, producing NaNs.

• AdamW Weight Decay Misconfiguration

  AdamW weight decay misconfiguration causes poor generalization or unstable training.

• AMP BF16 vs FP16 Confusion

  BF16 and FP16 mixed precision have different numerical properties; choosing the wrong one causes instability or wasted memory.

• Attention Softmax FP16 Overflow

  The attention scores are calculated as `Q @ K.T`. Before scaling by `1/sqrt(d_k)`, these dot products can be very large. If calculated purely in FP16, the dot product can exceed 65,504, causing an overflow to `inf`. Softmax on `inf` values produces `NaN`.

• Autograd Singularity at sqrt(0)

  The analytical derivative of sqrt(x) is 1 / (2 * sqrt(x)). If the input x evaluates to exactly 0, the denominator becomes 0, and the gradient evaluates to infinity. PyTorch's autograd engine will then propagate NaN back to the network's weights.

• Cosine Annealing Issue

  Cosine annealing issues arise from incorrect min_lr, T_max, or warm restarts that cause poor convergence.

• CrossEntropyLoss NaN on Pure Ignore-Index Batch

  When using `torch.nn.CrossEntropyLoss(ignore_index=-100, reduction='mean')`, if every target in the batch happens to equal `-100`, PyTorch calculates the sum of the valid losses (which is 0) and divides by the number of valid targets (which is also 0). This results in 0/0, which is NaN.

• CUDA Device-Side Assert from Tokenizer/Embedding Vocab Mismatch

  indexSelectLargeIndex: srcIndex < srcSelectDimSize assertion fires when input token ids exceed the embedding table size. The standard outcome of pairing a tokenizer that has added tokens with a model whose embeddings were never resized.

• CUDA Device-Side Assert Triggered

  A CUDA kernel hit a device-side assertion. Almost always an out-of-bounds index into an embedding, loss, or gather/scatter op. Because CUDA is asynchronous, the reported stack trace points at an unrelated later line, so the real cause is hidden unless you force synchronous execution.

• Curriculum Learning Issues

  Curriculum learning issues arise from poorly designed difficulty progression that hurts rather than helps training.

• Cyclic LR Scheduler Issues

  Cyclic LR scheduler issues arise from incorrect base/max LR, step size, or mode (triangular, triangular2, exp_range).

• DeepSpeed bf16 Gradient Norm Underflow

  DeepSpeed's bf16 training can trigger 'assert all_groups_norm > 0' because bf16 gradient norms can underflow to zero. This is a numerical precision issue specific to bf16's limited range. Denpex detects the underflow from DeepSpeed error logs and recommends the fix.

• DeepSpeed fp16: Current loss scale already at minimum

  Persistent fp16 overflow drives the dynamic loss scale down to its minimum and DeepSpeed aborts. Typically a bf16-pretrained model being fine-tuned in fp16 on hardware without bf16.

• DeepSpeed NaN from overlap_comm + contiguous_gradients

  Enabling both overlap_comm and contiguous_gradients in DeepSpeed ZeRO-3 causes gradient buffer reuse races that produce NaN losses. This is a known DeepSpeed bug where the communication overlap reads from a gradient buffer that is being rewritten by the contiguous gradient copy. Denpex detects the specific flag combination and recommends disabling one.

• DeepSpeed Training Hangs at Start with HuggingFace auto_find_batch_size

  DeepSpeed (ZeRO offload) training with HuggingFace Trainer auto_find_batch_size=True hangs right after model load at 0/N steps. The batch-size auto-search re-initializes the engine and desynchronizes collectives across ranks. Disable auto_find_batch_size for distributed runs.

• EMA Checkpoint Issue

  EMA (Exponential Moving Average) checkpoint issues cause problems with model averaging across checkpoints.

• EMA Decay Misconfiguration

  Exponential Moving Average (EMA) decay misconfiguration causes poor model averaging and training instability.

• EMA Decay Too High

  EMA (Exponential Moving Average) decay values that are too high or too low cause poor model averaging.

• Fine-Tuning Failure

  Fine-tuning failures occur when pretrained model knowledge is destroyed by aggressive learning rates or insufficient data.

• flash_attn_func produced all NaN values scale factor reciprocal overflowed FP8 E4M3 range

  The FP8 attention scale or its reciprocal left the representable E4M3 range, so the kernel produced non-finite output. This is a scale-calibration failure before it is a general training-divergence failure. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent numerics failures.

• FSDP Extra State Checkpoint Ignored

  A regression in PyTorch 2.3's Distributed Checkpoint (DCP) logic stripped the processing of the `_extra_state` dictionary during `set_model_state_dict`.

• FSDP Hang Due to Divergent Parameter Initialization

  FSDP requires all ranks to have identical model architectures and parameter shapes. If `param_init_fn` is used incorrectly (e.g., using random generation that affects tensor shapes or diverging control flow per rank), Rank 0 might have a different layer structure or shape than Rank 1. When FSDP tries to all-gather parameters, the metadata mismatches, causing NCCL to wait indefinitely for matching tensor sizes that never arrive.

• FSDP Mismatched Tensor Shapes Triggering NCCL Hang

  In FSDP, dynamic input shapes or uneven batch sizes across ranks can lead to mismatched tensor sizes during the AllGather operation for gradients or model parameters. NCCL expects the collective operation to use exact matching sizes across all ranks; otherwise, it hangs waiting for the expected byte count.

• Gradient Accumulation BatchNorm Issue

  Gradient accumulation with BatchNorm causes incorrect normalization because BN computes statistics on smaller sub-batches.

• Gradient Accumulation Misuse

  Gradient accumulation misuse causes incorrect gradient updates or memory issues.

• Gradient Clipping Missing

  Missing gradient clipping causes gradient explosion in RNNs, transformers, and GANs.

• Gradient Explosion

  Gradient explosion causes loss spikes, NaN loss, and training instability, especially in RNNs and deep networks.

• Gradient Explosion

  Gradient explosions cause loss to diverge to NaN. Denpex traces the norm spike to the layer and step.

• Gradient Explosion Caused by Corrupted All-Reduce (Not Learning Rate)

  Gradient norms explode suddenly while loss remains normal because the cross-rank reduction itself is corrupting values. In-network reduction (SHARP) faults, NCCL data corruption over a marginal link, or a mixed-precision comm dtype bug. Per-rank local gradients are healthy; only the aggregated result is wrong.

• GRPO/vLLM: 'Inference tensors cannot be saved for backward'

  GRPO training that feeds vLLM-generated tensors into the trainable graph fails with 'Inference tensors cannot be saved for backward'. vLLM produces tensors under torch.inference_mode(), which cannot enter autograd; clone/detach them at the generation→training boundary.

• Infinity Loss / Inf Weights

  Infinity loss or weights occur when loss values overflow to infinity, often from numerical instability in mixed-precision training.

• Init Seed Mismatch

  Different random seeds across runs cause non-reproducible results, making experiments hard to compare.

• Label Noise Training Issue

  Label noise in training data causes poor model generalization and unexpected training behavior.

• Layer Norm Weight Decay

  Applying weight decay to LayerNorm and bias parameters hurts training and can prevent convergence in transformers.

• LayerNorm backward pass produced Inf values input variance below machine epsilon

  LayerNorm produced non-finite gradients because the input variance underflowed. The backward pass divides by sqrt(var + eps); in FP16 a near-constant activation drives variance below representable range and the division explodes. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent numerics failures.

• Learned Temperature Collapse in Contrastive Loss

  In contrastive learning (like CLIP), logits are scaled by `1 / temperature`. If the temperature parameter is learned and not constrained, the optimizer may push it towards zero to artificially increase confidence and lower the loss. As temperature approaches 0, logits approach infinity. This causes `CrossEntropyLoss` (which computes exp) to produce infinities, yielding NaNs in the backward pass.

• Learning Rate Finder Result Misuse

  Misusing the results of a learning rate finder causes poor training setup choices.

• Lion Optimizer Issues

  Lion optimizer issues arise from incorrect learning rate (typically 3-10x lower than Adam) or momentum configuration.

• Lookahead Optimizer Issues

  Lookahead optimizer issues arise from improper k (inner steps) or alpha (slow weight) values that destabilize training.

• Loss Curve Anomaly

  Loss curve anomalies (spikes, plateaus, oscillations) indicate underlying training issues that need diagnosis.

• Loss is NaN gradients contained NaN in transformer layer mlp c_proj bf16

  Training loss became NaN. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent numerics failures.

• Loss Not Decreasing

  Loss not decreasing indicates fundamental training issues: wrong LR, broken model, bad data, or wrong loss function.

• Loss Plateau / Training Stalled

  Loss plateaus occur when training stops making progress, often due to suboptimal hyperparameters or model architecture issues.

• LR Too High

  Learning rate too high causes training instability, loss divergence, or NaN loss.

• LR Too Low

  Learning rate too low causes slow convergence, plateau at high loss, or training to appear stuck.

• LR Warmup-Decay Schedule Issue

  LR warmup-decay schedule issues cause training instability at transitions between phases.

• LR Without Warmup or Decay

  Training without learning rate warmup or decay causes slow convergence and poor final performance.

• Megatron-LM Hangs at Fused Kernel Compilation

  Megatron-LM training hangs at startup while compiling fused kernels (right after 'using torch.float16 for parameters ...') and never proceeds. A stale or contended megatron/fused_kernels/build directory blocks the ninja build; deleting it (or pre-compiling once) unblocks the run.

• Missing GradScaler Update in AMP Loop

  When using `torch.cuda.amp.autocast()`, gradients are computed in float16. The dynamic range of float16 is narrow, causing gradients to overflow to infinity. If a `GradScaler` is not used, or if `scaler.update()` is omitted, the scale factor isn't adjusted to prevent these overflows, leading to NaN gradients and subsequently NaN loss.

• Mixed Precision Loss Scale

  Mixed precision loss scale issues occur when GradScaler doesn't update properly, causing underflow or overflow in FP16 training.

• Mixed Precision Overflow

  Mixed precision overflow occurs in FP16 training when values exceed FP16 range (65504), causing inf/NaN.

• MoE Activation Recomputation Export OOM

  The conversion script loads the optimizer states into VRAM alongside the model parameters. For dense models, this fits. For MoE models trained with activation recomputation, the sheer parameter volume makes simultaneous optimizer loading mathematically impossible on a single GPU.

• MoE Pipeline Stage Asymmetry Hang

  When using a custom `--pipeline-model-parallel-layout`, users may configure a pipeline stage to have zero MoE layers (e.g., placing all MoE layers in the middle stages). However, the logging/loss aggregation function `reduce_aux_losses_tracker_across_ranks` implicitly assumes all ranks in the pipeline group possess MoE layers and invokes an AllReduce across the entire pipeline group. The stage with 0 MoE layers bypasses the MoE loss logic and never joins the AllReduce, causing the other stages to hang forever waiting for it.

• NaN Detection and Skip

  NaN detection and skipping prevents corrupted gradients from propagating but requires careful implementation.

• NaN Loss

  NaN loss in training is critical because it propagates through all parameters and corrupts the model permanently.

• NaN Loss During Training

  NaN loss corrupts training state. Denpex traces NaN propagation to the originating layer.

• NaN Loss from TransformerEngine FP8 Scaling Saturation

  FP8 training with delayed scaling produces NaNs when an activation outlier saturates the scaling factor (amax history too short / margin too small). Loss and gradients are healthy until the exact step the scale saturates. Distinguishing it from LR- or data-driven NaN.

• One-Cycle Policy Issue

  One-cycle policy issues arise from incorrect max_lr, momentum range, or training duration that destabilize training.

• Polyak Averaging Issues

  Polyak averaging issues arise from incorrect averaging window, frequency, or weight initialization.

• PyTorch JIT Compile Error

  PyTorch JIT compilation errors crash training when the script or model uses unsupported features.

• PyTorch Lightning + torch.compile: MisconfigurationException on self.log

  Training a Lightning model with torch.compile crashes in the logger connector with a 'called self.log twice with different arguments' MisconfigurationException raised inside the compiled region. Dynamo traces through Lightning's result collection and the metadata-equality branch raises instead of breaking the graph. Insert a graph break or compile only the model forward.

• PyTorch torch.compile Error

  torch.compile errors crash training when the compiled graph has issues with the model or hardware.

• Ranger Optimizer Issues

  Ranger optimizer combines RAdam and Lookahead; misconfiguration can cause training instability or poor convergence.

• RMSNorm Variance Overflow in FP16

  In RMSNorm, the sum of squared activations is computed. In large models (e.g., hidden dimension > 4096), summing the squares of fp16 activations can easily exceed 65,504 (the maximum fp16 value), causing an overflow to infinity. The subsequent step computes `1.0 / sqrt(inf)`, which results in 0. The activations are zeroed out, killing gradients and causing NaNs upon backpropagation.

• Root FSDP Activation Checkpointing Backward Assert

  Nesting all parameters deep within child FSDP modules leaves the outer FSDP root empty. When activation checkpointing triggers recomputation, the lack of parameters in the root module desynchronizes the FSDP state machine during the backward hooks.

• Runtime Error (Generic)

  Generic runtime errors can indicate various issues from code bugs to hardware problems.

• SAM (Sharpness-Aware Minimization) Optimizer Issues

  SAM optimizer issues arise from the two-forward-pass requirement, learning rate issues, or rho misconfiguration.

• SGD Momentum Configuration Issue

  SGD momentum misconfiguration causes training to oscillate, diverge, or converge slowly.

• Spectral Norm Clipping

  Spectral norm clipping for GAN training has different requirements than gradient clipping for other models.

• Spectral Normalization Collapse

  Spectral normalization can cause mode collapse or training failure when applied incorrectly to GANs.

• SWA (Stochastic Weight Averaging) Training

  SWA training issues arise from incorrect averaging frequency, learning rate schedule for SWA, or BN update steps.

• Transformer Divergence Without Gradient Clipping

  Transformer models are highly sensitive to activation magnitude growth across layers. A high learning rate or lack of layer normalization (e.g. post-LN without warmup) causes gradients to accumulate multiplicatively during backpropagation. This exceeds the dynamic range of FP16/FP32, resulting in an overflow to infinity.

• Unsloth Fused Loss Breaks with Transformers average_tokens_across_devices=True

  Multi-GPU fine-tuning with Unsloth breaks when the Transformers default average_tokens_across_devices=True multiplies the loss by num_processes. The Unsloth fused-loss backward does not expect that scaling and the loss tensor is corrupted (becomes an int / loses grad). Setting average_tokens_across_devices=False fixes it.

• Unsloth Gemma 3: 'Gemma3ModelOutputWithPast' object has no attribute 'loss'

  Fine-tuning Gemma 3 with Unsloth crashes with AttributeError: 'Gemma3ModelOutputWithPast' object has no attribute 'loss'. A transformers change to the Gemma3 output class outran Unsloth_zoo's patch, which still read outputs.loss. Upgrading unsloth/unsloth_zoo (and pinning a compatible transformers) fixes it.

• Unsloth/TRL Warning: attention implementation not flash_attention_2 with packing

  When fine-tuning with Unsloth + TRL using sample packing or padding-free training, the trainer warns that the attention implementation is not flash_attention_2 even though it was configured. Flattened batches can behave incorrectly under non-FA2 attention; the warning came from Unsloth's compiled trainer not propagating attn_implementation and was fixed in a release.

• ValueError: Another Profiling Tool Is Already Active

  Enabling PyTorch Lightning's profiler raises 'ValueError: Another profiling tool is already active'. Python's cProfile allows only one active profiler per thread, so a second cProfile-based profiler (coverage, a debugger, a nested AdvancedProfiler) collides. Run only one cProfile-based tool at a time.

• Warmup Missing

  Missing warmup causes early training instability, especially with large learning rates or transformer models.

• Weight Decay Misconfiguration

  Weight decay misconfiguration causes overfitting, underfitting, or training instability depending on the wrong choice.

• Weight Divergence Across Ranks

  Weight divergence silently corrupts distributed training. Denpex detects divergence by comparing per-rank weight snapshots.

• Wrong Weight Initialization

  Wrong weight initialization causes training instability with NaN losses, slow convergence, or dead neurons.

• Zero Gradient / Dead Neurons

  Zero gradients stall training when ReLU neurons die or gradient flow is broken in the network.

• ZeRO-3 Partitioned Checkpoint Dimensionality Collapse

  The parameters were partitioned across GPUs using ZeRO Stage 3, and the checkpoint saving mechanism did not consolidate them back to the universal format. The saved state_dict contains only the local shard of the root rank.

• ZeroDivisionError float division by zero GradScaler.step all gradients unscaled to zero or Inf

  The AMP gradient scaler reached a state where every gradient was zero or non-finite, so the scale update divided by zero. The scaler is reporting a numerical collapse upstream, not causing one. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent numerics failures.