Distributed Training error references

Browse 67 published references for distributed training failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• Accelerate Checkpoints Miss Weights When Using prepare_model Instead of prepare

  Model weights are not written by accelerator.save_state when the model was set up with accelerator.prepare_model(model) instead of accelerator.prepare(...). prepare_model only wraps the model for distribution; it does not register it for checkpointing.

• Accelerate-DeepSpeed Watchdog Timeout Mismatch

  Hugging Face Accelerate integration drops the `InitProcessGroupKwargs` timeout parameter when DeepSpeed owns the distributed initialization process. The PyTorch watchdog defaults back to 600 seconds.

• Asymmetric PT2 Compilation Collective Desync

  Data-dependent control flow causes the JIT compiler to generate divergent execution graphs across ranks based on local data variance. Ranks attempt to execute different collective operations.

• bitsandbytes 'invalid configuration argument' (ops.cu) with DeepSpeed Offload

  Combining a bitsandbytes 8-bit optimizer with DeepSpeed ZeRO offload crashes with 'Error invalid configuration argument at line 216 in file .../bitsandbytes/csrc/ops.cu'. The two optimizer-state managers conflict over offloaded/partitioned parameters; disabling ZeRO offload (zero_stage 0) or dropping the 8-bit optimizer resolves it.

• Checkpoint Save/Restore Gathering Timeout

  NCCL timeout occurs while gathering the state dictionary or broadcasting during a checkpoint save or restore operation.

• Cowardly refusing to serialize non-leaf tensor

  Something still attached to the autograd graph was handed to the checkpoint writer. Saving it would serialise the graph as well as the values, so torch refuses rather than writing a checkpoint that cannot be loaded cleanly. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent pytorch-fsdp-ddp failures.

• DDP Hang at Epoch Boundary

  DDP training hangs at the end of an epoch when one rank exhausts its data partition before others.

• DDP Hang Due to Unused Parameters in Forward Pass

  When find_unused_parameters=False (the default), DDP assumes every parameter in the model will receive a gradient. If a parameter is skipped in the forward pass (e.g., due to a conditional if-statement), its gradient is never computed. DDP waits indefinitely in the backward pass for the all-reduce operation on that missing gradient, deadlocking the training step.

• DDP Hang from Uneven Dataset Sizes Across Ranks

  When using DistributedSampler without drop_last=True, the dataset size might not be perfectly divisible by the number of GPUs. Consequently, some ranks might have N batches, while others have N-1. The ranks that finish early exit the training loop, while the remaining ranks enter the next batch and block indefinitely on dist.all_reduce waiting for the finished ranks.

• DDP NaN Detected on One Rank

  DDP NaN detection on a single rank often indicates inconsistent data or hardware between ranks.

• DDP OOM During AllReduce

  DDP OOM during allreduce occurs when NCCL communication buffers exceed available GPU memory.

• DDP Port Conflict

  DDP port conflicts occur when the chosen master port is already in use by another process.

• DDP Rank Stuck During Training

  One DDP rank becomes unresponsive and stalls all ranks waiting for gradient sync.

• DDP Setup Error

  DDP setup errors prevent distributed training from initializing correctly.

• DeepSpeed 0.14.x Regression: 'Expected all tensors to be on the same device' (ZeRO-3 / Adam Offload)

  After upgrading to DeepSpeed >0.14.0 (e.g. 0.14.2), ZeRO-3 / optimizer-offload training crashes with 'Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!'. A regression in the 0.14.x ZeRO-3 path left some tensors on CPU; upgrading past the fix (or downgrading) resolves it.

• DeepSpeed Initialization Failed

  DeepSpeed initialization fails when configuration is invalid or incompatible with the model.

• DeepSpeed Initialization Failed

  DeepSpeed initialization fails when configuration is invalid or incompatible with the model.

• DeepSpeed MoE + ZeRO-3 Hang. Missing Leaf Module Marking

  DeepSpeed ZeRO-3 with Mixture-of-Experts (MoE) models hangs during training because MoE blocks need to be marked as leaf modules for ZeRO-3 parameter gathering. Without this marking, ZeRO-3 attempts to gather parameters recursively through the MoE router, causing deadlocks. Denpex detects the MoE+ZeRO-3 hang pattern and recommends the leaf module fix.

• DeepSpeed Out of Memory

  DeepSpeed OOM errors crash ZeRO-optimized training when memory optimization settings don't match model architecture.

• DeepSpeed Pipeline Parallel Hangs with Variable Input Shapes

  DeepSpeed pipeline-parallel training (PipelineModule) hangs mid-run when microbatch input shapes vary. The PP engine caches the first microbatch's tensor shapes for point-to-point send/recv buffers, so a later differently-shaped microbatch mismatches the buffers and deadlocks. Pad to a fixed shape or upgrade to a release with dynamic-shape support.

• DeepSpeed PPO Actor forward pass deadlocked waiting for Critic ZeRO-3 parameter gather

  The actor blocked waiting for a ZeRO-3 parameter gather owned by the critic. Two ZeRO-3 models in one process share a parameter coordinator; if their forward passes interleave, each can wait on a collective the other has not entered. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent rlhf failures.

• DeepSpeed ZeRO Stuck / Hang

  DeepSpeed ZeRO hangs stall training when gradient synchronization stalls or parameter offload produces deadlocks.

• DeepSpeed ZeRO-2 tensor bucket buffer exhausted during reduce_scatter

  ZeRO-2 batches gradients into a fixed communication bucket before reduce-scatter. A gradient tensor larger than the bucket, or too many arriving at once, exhausts it and the reduction cannot proceed. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent deepspeed failures.

• DeepSpeed ZeRO-3 + PyTorch 2.5 _parameters Dict Error

  DeepSpeed ZeRO-3 crashes with 'dict object has no attribute _in_forward' when used with PyTorch 2.5+, which changed the internal _parameters attribute from a dict to a different type. Denpex detects the version incompatibility and recommends the correct DeepSpeed or PyTorch version.

• DeepSpeed ZeRO-3 Error with Parameters of Multiple Dtypes (allgather)

  DeepSpeed ZeRO-3 errors during parameter allgather when the model holds parameters in more than one dtype (e.g. bf16 base with some fp32 modules). Older ZeRO-3 assumed a single dtype for coalesced allgather buffers; upgrade to a release that supports multiple dtypes, or unify the parameter dtype.

• DeepSpeed ZeRO-3 Small Parameter Partition Bug

  DeepSpeed ZeRO-3 crashes with UnboundLocalError in partition_parameters.py when the model has very small parameters (fewer elements than the number of GPUs). The partition math divides a small parameter across more GPUs than it has elements. Denpex detects the small-parameter partition failure and recommends the configuration fix.

• DeepSpeed ZeRO-Offload NVMe aio_write failed: No space left on device

  DeepSpeed ZeRO-Offload filled the NVMe filesystem used for optimizer or parameter swap. The failed aio_write means the offload state is incomplete, even if the training process reports a later optimizer error. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent deepspeed failures.

• dist.all_gather Timeout from Dynamic Tensor Shapes

  dist.all_gather requires every participating rank to pass a tensor of the exact same dimensions. If Rank 0 detects 5 bounding boxes (shape [5, 4]) and Rank 1 detects 3 bounding boxes (shape [3, 4]), the underlying NCCL backend gets mismatched buffer sizes. Rather than throwing an immediate shape error, NCCL hangs indefinitely, waiting for matching sizes.

• DistributedDataParallel raises an internal reducer assertion when a backward pass arrives unexpectedly

  DDP's gradient reducer arms itself for exactly one backward pass per forward and disarms once that pass is finalised. A second backward, or a backward whose forward was run under a different synchronisation context, arrives when the reducer is not armed and trips an internal assertion whose text invites you to file a PyTorch bug.

• Finding the Culprit Rank in an NCCL Hang

  When one rank stalls, NCCL's synchronous collectives freeze every rank, masking which GPU actually failed. Localizing the culprit needs divergence signals, not the NCCL stack trace.

• FSDP All-Gather Timeout

  FSDP all-gather timeouts stall training when sharded parameters cannot be collected from distributed ranks.

• FSDP Failed to allocate unpartitioned full weights during forward pre-hook

  FSDP ran out of memory gathering a layer's full weights for the forward pass. Sharding reduces STEADY-STATE memory, but each layer is briefly unsharded to compute, so the peak is set by the largest single layer, not by the sharded average. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent pytorch-fsdp-ddp failures.

• FSDP fails when composed with CPU parameter offloading

  Sharding and CPU offloading both move parameters, and each assumes it is the component deciding where a parameter lives. Enabling them together produces a failure during setup or the first step, in a stack that works correctly with either one alone.

• FSDP Flat Parameter Error

  FSDP flat parameter errors occur when the flattened parameter management fails during sharding or unsharding operations.

• FSDP Missing Keys / Unexpected Keys

  FSDP state_dict missing keys errors prevent checkpoint saving or loading in sharded models. Denpex identifies the source of key mismatches.

• FSDP Mixed Precision Error

  FSDP mixed precision errors arise when parameter precision settings conflict between FSDP wrapping and autocast.

• FSDP preparation fails because activation checkpointing calls a wrapping policy that was never set

  Choosing not to auto-wrap is legitimate and leaves the wrapping policy unset. Activation checkpointing then tries to apply that policy to decide which submodules to checkpoint, calls the empty value, and preparation fails with a type error naming nothing that appears in the configuration.

• FSDP reduce_scatter backward hook timed out waiting for gradient sync

  An FSDP gradient reduce-scatter did not complete within the collective timeout. Like any collective timeout this names the ranks that were WAITING, not the one that failed to arrive, one rank left the collective early or died, and the rest report the timeout. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent pytorch-fsdp-ddp failures.

• FSDP+QLoRA ValueError: Must flatten tensors with uniform dtype (float32 vs bfloat16)

  FSDP combined with QLoRA fails to build its flat parameter: 'Must flatten tensors with uniform dtype but got torch.float32 and torch.bfloat16'. QLoRA keeps some parameters in float32 while the base is bfloat16; FSDP requires one dtype per flattened parameter group. Use the maintained FSDP+QLoRA config and consistent dtypes.

• FSDP2 raises a KeyError on a tied weight while sharding the model

  A language model whose output projection shares its tensor with the input embedding fails to shard. When fully_shard rebuilds parameters as DTensors, the shared tensor appears once rather than twice, the mapping from old parameters to new ones loses the tied name, and preparation dies on a KeyError naming a weight that plainly exists in the model.

• FSDP2 Unwrapped Model Still Has DTensor Weights, save_pretrained Fails

  After unwrapping an FSDP2 (fully_shard) model the parameters are still DTensors, so save_pretrained or a plain state_dict produces invalid storage or sharded tensors. You must gather a full, unsharded state dict before saving and write only on rank 0.

• jax.distributed initialize timed out waiting for rank 0 on coordinator

  jax.distributed.initialize() timed out waiting for all processes to check in with the coordinator. One or more workers never reached the rendezvous, they crashed at startup, cannot reach the coordinator address, or were given the wrong process count. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent jax failures.

• jax.errors.XlaRuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate

  XLA could not allocate device memory. JAX preallocates ~75% of VRAM at first use by default, so an OOM here often reflects the preallocation policy or memory fragmentation across recompilations rather than a genuinely oversized model. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent jax failures.

• Megatron Sequence Parallel requires Tensor Parallel size > 1

  Sequence parallelism was enabled while tensor parallel size remained 1. Megatron partitions sequence work across the tensor-parallel group, so there is no group to partition across in this configuration. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent megatron failures.

• Megatron-LM Initialization Failed

  Megatron-LM initialization errors prevent large model training from starting due to configuration mismatches.

• MXFP4 expert weights lose their scale attribute when a mixture-of-experts model loads under FSDP2

  A quantised mixture-of-experts checkpoint loads on a single device and fails under sharded loading. Materialising a module from the placeholder device reads each parameter by name, and the quantised expert block does not expose the scale attribute the checkpoint expects, so loading stops on a missing attribute with a helpful suggestion naming a different one.

• NCCL Broadcast Hang

  NCCL broadcast operations can hang when one rank fails to participate in the collective.

• NCCL Watchdog Configuration

  NCCL watchdog settings control how long collectives can stall before timing out. Wrong settings cause premature or delayed failure detection.

• Node Rank Mismatch at NCCL Init

  The configured world size does not match the number of ranks that actually connect, so NCCL initialization hangs or errors during the communication-setup stage.

• Overflow detected during fp16 gradient unscaling. Skipping step, loss scale halved

  FP16 gradient overflow forced the loss scaler to halve repeatedly. Occasional skipped steps early in training are normal; a scale that keeps collapsing means gradients are genuinely exploding or the model is numerically unstable in FP16. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent deepspeed failures.

• P2PTimeoutError: Timeout receiving forward activations from rank

  A pipeline stage waited for activations from its neighbour and timed out. In pipeline parallelism a single slow or dead stage stalls the whole pipeline, so the rank that reports the timeout is the VICTIM, the fault is upstream, in the stage that failed to send. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent megatron failures.

• PlacementGroupCreationError placement group creation timed out insufficient GPU resources

  Ray could not reserve the requested placement group before the timeout. The cluster does not have a set of nodes that simultaneously satisfies the bundle layout, commonly an RLHF setup asking for colocated vLLM actors and trainer workers on the same node. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent rlhf failures.

• PPOTrainer CUDA out of memory cannot fit reference model policy model and value head

  TRL PPO could not keep the policy, reference model, value head, activations, and optimizer state within the available VRAM. The simultaneous model residency makes this different from a single-model batch OOM. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent rlhf failures.

• PyTorch CPU and CUDA Device Mismatch: Find the Operand

  Locate the mismatched input, parameter, buffer, index or restored optimizer state at the failing PyTorch operation and verify the original workload.

• PyTorch Distributed Error

  PyTorch distributed errors prevent DDP and FSDP from functioning correctly.

• Ray client connection aborts on ARM64 after a grpcio upgrade while x86 is unaffected

  Connecting to a cluster starts failing intermittently on ARM64 nodes after a transport library is upgraded, while the identical image on x86 continues to work. The client reports that the server aborted initialisation, and the server's own reason is written to a separate file the client never shows.

• Reference model logits shape does not match policy logits shape vocab mismatch

  The reference model and the policy model have different vocabulary sizes, so their logits cannot be compared. DPO scores the policy against the reference token by token, a vocab difference of even a few entries makes every comparison meaningless. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent rlhf failures.

• Rotary embedding dimension must be divisible by head dimension

  The rotary embedding dimension does not divide evenly into the attention head dimension. This is a configuration arithmetic error, caught at construction: hidden_size / num_attention_heads must be compatible with the rotary percentage. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent megatron failures.

• Silent Cascading Straggler

  A single GPU drops in performance without throwing an error, causing all other GPUs in the collective to wait. This silently degrades cluster-wide throughput.

• SyncBatchNorm Deadlock During Single-Rank Evaluation

  SyncBatchNorm performs an all-reduce across all processes to calculate global batch statistics. If the script only runs validation on rank 0 (a common pattern to save time), rank 0 will hit the SyncBatchNorm layer and block forever waiting for ranks 1-N, which are sitting idle. SyncBatchNorm communicates even in eval mode if not explicitly handled.

• Tensor Parallel Error

  Tensor parallel errors occur when model layer splitting across GPUs fails due to dimension mismatches or communication issues.

• torch.distributed.elastic.multiprocessing.errors.ChildFailedError exitcode -9

  torchrun ChildFailedError, a worker process died and torchelastic is reporting the WRAPPER error. The real failure is in the per-rank traceback above/below this block (exitcode -9 = OOM kill, 1 = python exception, -11 = segfault). This entry explains how to confirm the cause, apply the fix, and separate it from adjacent pytorch-fsdp-ddp failures.

• torch.OutOfMemoryError: CUDA out of memory. Tried to allocate

  GPU ran out of memory during training. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent pytorch-fsdp-ddp failures.

• Torchrun and SLURM Job Step Incompatibility

  Torchrun's approach to spawning jobs conflicts with SLURM's process management, resulting in hung NCCL initialization, rank mismapping, and job allocation failures.

• Triaging torchrun ChildFailedError: the Wrapper Is Never the Root Cause

  ChildFailedError is torchelastic's envelope around a dead worker. The exitcode/signal in its table is the actual signal: -9 host OOM-kill, -11 segfault, -6 abort (often CUDA), 1 python exception whose traceback sits in that rank's own log.

• Watchdog caught collective operation timeout WorkNCCL AllGather

  NCCL collective timed out. One rank stopped participating. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent pytorch-fsdp-ddp failures.

• ZeROStage3ParamStatusException: Parameter in Partitioned state, expected Available

  ZeRO-3 keeps parameters sharded until they are gathered for use. Code touched a parameter outside a gather context, so it was still partitioned, typically custom forward logic, weight tying, or an initialization step that reads .data directly. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent deepspeed failures.