Performance error references

Browse 10 published references for performance failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• An inference scheduler deadlocks under sustained load and stops issuing work

  A continuous-batching scheduler admits requests against a token budget it must not exceed. Under sustained concurrency the accounting can reach a state where no admitted request can advance and no new one can be admitted, so the server stops issuing work while remaining connected and apparently healthy.

• CPU Affinity Misconfiguration

  CPU affinity misconfiguration prevents DataLoader workers and training threads from using optimal CPU cores.

• DeepSpeed ZeRO-3 Slow (Synchronous Param Prefetch)

  ZeRO-3 throughput collapses versus ZeRO-2 because the parameter prefetch behaves synchronously, so transfer time fails to overlap with compute.

• GPU Utilization Low

  Low GPU utilization indicates training is bottlenecked by data loading, CPU work, or communication.

• HBM3 Memory Failures on 16384-GPU Cluster During Llama 3 Training

  During the 54-day pre-training of Llama 3 405B on 16,384 NVIDIA H100 GPUs, 419 unexpected interruptions occurred at an average rate of one every 3 hours. HBM3 memory failures alone accounted for 17.2% of all unplanned interruptions, while total GPU-related failures (including NVLink issues) comprised 58.7%. Silent data corruption caused 6 additional undetected job interruptions.

• Persistent Straggler Rank from Single-GPU Thermal Throttling

  One rank consistently 2-4x slower drags every synchronous collective. When the cause is a failed fan/blocked airflow on exactly one GPU, the fleet-wide symptom (slow allreduce everywhere) hides a single-node hardware root cause.

• torch.compile AOTAutograd Graph Compilation Timeout

  During AOTAutograd phase of torch.compile, complex dynamic control flow or deeply nested autograd graphs can cause the Inductor compiler to stall indefinitely. This often manifests as a single node hanging at 100% CPU utilization while peer nodes wait in NCCL collectives, eventually triggering cluster-wide timeout.

• Uneven Pipeline Stage Partitioning as Primary Straggler Cause in LLM Training

  In a five-month trace analysis of ByteDance's LLM training cluster, uneven pipeline stage partitioning was identified as the most prevalent cause of training stragglers, affecting 39.3% of jobs. The last pipeline stage containing loss computation layers is significantly heavier than earlier stages, creating systematic imbalance that propagates cascading bubbles through the entire pipeline.

• Unsloth Qwen3-30B-A3B MoE Fine-Tuning Extremely Slow / Low GPU Utilization

  Fine-tuning the Qwen3-30B-A3B MoE model with Unsloth is far slower (200-300 s/step) at 10-20% GPU utilization than the comparable dense Qwen3-32B, which runs at full utilization. Missing/unoptimized MoE expert-routing kernels in that Unsloth version starved the GPU; upgrading restores throughput.

• vLLM throughput collapses as the scheduler preempts and recomputes the same requests

  When KV cache blocks run out, the vLLM scheduler evicts in-flight sequences and recomputes them later. Under sustained load the recomputation consumes the cache and compute that would have retired other requests, so evictions beget evictions and a large share of GPU time goes to work that has already been done once.