Reliability error references

Browse 42 published references for reliability failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• 1-2 GPU Hardware Failures Per Week During BLOOM 176B Training on 384 A100 GPUs

  During the 4-month training of the 176B-parameter BLOOM model on 384 NVIDIA A100 GPUs, the team experienced 1-2 GPU hardware failures per week on average. Each failure required a 30-60 minute node replacement and checkpoint rollback. The cluster used 48 nodes with 32 spare GPUs available for hot-swap replacement.

• A worker loses its GPU mid-run and the job hangs until a collective times out

  When a GPU disappears from its host, a bus fault, a driver reset, a thermal or power event, the rank owning it stops answering. Its peers are inside a collective waiting for data that will never arrive, so the job does not crash at the moment of failure. It stalls, and the first error anyone sees is a timeout on a healthy rank several minutes later.

• Automatic Node Failure Detection and Training Resumption Using Node Doctor and Watchdog

  MosaicML's training platform automatically detected hardware failures and resumed training without human intervention using its Node Doctor and Watchdog services. During Stable Diffusion training on 128 A100 GPUs, these systems eliminated the need for 24/7 human babysitting by automatically capturing failure signals, draining failed nodes, and resuming from the last checkpoint. This pattern was critical for achieving production reliability at sub-$50k training costs.

• Cgroup Memory Limit

  Cgroup memory limits cause OOM kills when training exceeds the container or pod's memory limit.

• Checkpoint Partial Save

  Partial checkpoint save happens when training crashes or is killed before all model state is saved.

• DataLoader Failure

  DataLoader failures cause training to crash or hang when data loading is misconfigured or data is corrupted.

• DeepSpeed Inference Degrades Generation Quality (GPT-NeoX/Pythia kernel injection)

  GPT-NeoX/Pythia models produce degraded or garbled text under DeepSpeed Inference with kernel injection, while the same model generates correctly without DeepSpeed. The injected fused kernels for the NeoX architecture had correctness bugs in some versions. Disable kernel injection or pin a known-good DeepSpeed version.

• DeepSpeed ZeRO-3 NVMe Offload Checkpoint Race (FileExistsError)

  All ranks race to write into the same consolidation directory during ZeRO-3 NVMe-offload checkpoint save, so only rank 0 succeeds and the rest hit FileExistsError.

• DeepSpeed ZeRO-3: 'weight' must be 2-D at F.embedding

  After ZeRO-3 training, accessing a partitioned parameter directly (e.g. an embedding weight) during inference sees a flattened 1-D tensor and raises a shape error.

• DeepSpeed ZeRO-3: Cannot partition a param in flight (save)

  save_16bit_model() under ZeRO-3 asserts because parameters are still in flight when the save interval is not a multiple of the gradient-accumulation steps.

• DeepSpeed ZeRO-3: still have inflight params (backward)

  ZeRO-3 backward fails because parameters from a previous forward path remain INFLIGHT, common with dynamic forward graphs such as RLHF, NAS, or conditional branching.

• Disaster Recovery Missing

  Missing disaster recovery planning means training progress can be lost due to hardware failure, corruption, or natural disaster.

• GB200 NVL72 driver hang with Xid 150 and 154 after long uptime is a firmware bug, not a dead GPU

  On GB200 NVL72 racks running R580 or later drivers, NVLink task scheduling can hang the GPU driver after roughly sixty days of uptime and emit Xid 150 and Xid 154. The cause is an overflow in the NVLink management microcode in the vBIOS, so replacing the card does not fix it.

• Graceful Shutdown Missing

  Missing graceful shutdown handlers cause training jobs to be killed without saving final checkpoint or metrics.

• HDF5 Corruption

  HDF5 corruption occurs when files are not properly closed, written from multiple processes, or have incompatible versions.

• Job Preemption / Spot Instance Termination

  Preemption kills cloud spot instances mid-training, losing progress if checkpoints aren't saved frequently enough.

• Mid-Day Temperature Fluctuations Causing 1-2% Training Throughput Variation

  During Llama 3 training on 16,384 H100 GPUs, Meta observed that mid-day temperature fluctuations of 5-10 degrees Celsius caused 1-2% throughput variation across the training cluster. The effect was attributed to GPU dynamic voltage and frequency scaling responding to ambient temperature changes, with the entire cluster's power consumption fluctuating by tens of megawatts throughout the diurnal cycle.

• Missing Signal Handler

  Missing signal handlers prevent graceful shutdown of training, causing checkpoint loss and resource leaks.

• MTTF Optimization from 0.33 to 3.66 Days Through Purpose-Built Infrastructure Architecture

  CoreWeave's six-week benchmark training a 30B-parameter Llama-3-style model on 1,024 H100 GPUs demonstrated that purpose-built infrastructure architecture can improve Mean Time to Failure from the industry baseline of 0.33 days to 3.66 days, a 10x improvement. Key architectural differences included bare-metal GPU access, dual-fabric networking, topology-aware scheduling via SUNK, and asynchronous checkpointing.

• MTTF Scaling Inversely with GPU Count in Large ML Research Clusters

  Meta FAIR's analysis of 11 months of operational data across two Research SuperClusters confirmed that Mean Time to Failure for training jobs scales inversely with GPU count, dropping from 47.7 days at 8 GPUs to approximately 7.9 hours at 1,024 GPUs. At 16,384 GPUs, the projected MTTF is 1.8 hours, making hardware faults a near-certainty during any multi-day training run.

• Network Partition

  Network partitions split distributed training into isolated groups, causing hangs, deadlocks, or wrong gradients.

• NFS Mount Failure

  NFS mount failures during training cause silent data corruption, hangs, or training failures when accessing shared storage.

• NIXL Firmware Page Registration Fan-Out Triggers Host OOM Kills on HGX H200 and B200

  NVIDIA Inference Xfer Library workloads on HGX H200 and HGX B200 instances triggered host OOM kills because UCX registered GPU memory as firmware pages across all 8 NICs simultaneously, consuming ~34 GiB of host memory per GPU in firmware-level registrations. The root cause was traced to UCX registering every reachable HCA for each buffer, creating firmware page fan-out proportional to NIC count.

• OOM Killed Mid-Step

  OOM kills during training are abrupt and lose progress, often happening mid-step when peak memory is reached.

• Python Garbage Collection Triggering Periodic Training Stragglers in Distributed LLM Training

  ByteDance's production trace analysis found that Python's automatic garbage collection causes periodic, transient straggler behavior in distributed LLM training. Full GC passes on long-lived objects (preallocated buffers, tokenizer state, lookup tables) can stall rank computation for 200-500ms per event, enough to delay NCCL collectives and create cascading pipeline bubbles.

• PyTorch Caching Allocator Memory Fragmentation Causing False Straggler Slowdown

  ByteDance's production straggler analysis identified PyTorch's CUDA caching allocator memory fragmentation as a previously unreported cause of training stragglers. Over long-running jobs, the allocator fragments GPU memory into small non-contiguous blocks, causing kernel launches that require large contiguous allocations to trigger expensive defragmentation cycles on specific ranks.

• Replayed/Archived Logs Triggering False Hardware Diagnoses

  Postmortem archives, log-replay pipelines, and documentation snippets containing old Xid/ECC lines get concatenated into live streams and pasted into diagnostic tools, producing confident "replace the GPU" verdicts on healthy fleets. A diagnosis engine must timestamp-gate hardware evidence.

• Sequence Length Imbalance Causing Distributed Training Stragglers

  Sequence length imbalance across micro-batches within the same global batch causes certain pipeline stages to compute significantly longer than others. ByteDance's production trace analysis found that 21.4% of straggler-affected jobs showed measurable throughput degradation due to variable-length sequences creating uneven compute load across data-parallel ranks.

• Silent Data Corruption

  Silent data corruption during training causes wrong results without error messages, often from hardware issues.

• Silent Data Corruption from GPU Hardware Faults Causing Loss Spikes and Model Divergence

  Silent data corruption arising from latent GPU hardware defects bypasses ECC and other hardware detection mechanisms, causing incorrect computation results without any error signal. In LLM training, these faults manifest as sudden loss spikes, NaN propagation, or gradual parameter divergence that may go undetected for thousands of steps. Google reported SDC events weekly to biweekly during Gemini training, and Meta attributed 6 unplanned interruptions to SDC during Llama 3 training.

• Silent Hang / Unresponsive Training

  Silent hangs stall training without error messages. Denpex heartbeat detection identifies stuck ranks.

• Single-Bit Exponent Flip in BF16 Causing Silent Gradient Divergence Across Data-Parallel Ranks

  In BF16 distributed training, a single-bit flip in the exponent field of a gradient tensor can propagate silently through NCCL all-reduce without detection, causing one rank's gradient update to diverge from all others. Research shows that high-order bit flips (exponent bits 1-3) are detected with 99% accuracy using Wasserstein divergence metrics, but undetected flips cause permanent model divergence within 10-50 steps.

• Spot Instance Preemption

  Spot instance preemption can kill training jobs with little notice, causing data loss and incomplete training runs.

• Straggler / Slow-Rank Detection

  One rank runs slower than the rest, so every collective waits on it. There is no crash and no error. Just falling throughput and idle GPUs. Stragglers are the dominant cause of silent efficiency loss at scale and must be detected by comparing per-rank step times, not by waiting for a timeout.

• Training Checkpoint Corruption on Write

  Training checkpoint corruption on write happens when storage failures, power loss, or process kills interrupt the write operation.

• Training Restart Stuck

  Training restarts after failure can get stuck if cleanup didn't complete properly, blocking new training jobs.

• Training Resume Failure

  Training resume failures occur when checkpoints can't be loaded properly to continue from a previous run.

• Training Stuck No Progress

  Training appears stuck with no progress, often caused by deadlocks, infinite loops, or network hangs in distributed training.

• vLLM returns 500 on every request after the EngineCore process dies

  The vLLM V1 API server survives while the separate EngineCore subprocess that owns the model does not. Once that child is gone the server keeps accepting connections and answers every one of them with an internal error until it is restarted, so an outage presents as a healthy-looking process serving nothing.

• Wandb/TensorBoard Failure

  Weights & Biases or TensorBoard failures cause loss of experiment tracking, making debugging and comparison hard.

• Zombie GPU Process / Orphaned Job

  Zombie processes hold GPU memory after the main job exits. Denpex auto-detects and kills them.

• Zombie Process

  Zombie processes occur when child processes aren't properly reaped, exhausting process table and file descriptors.