ML Training Failure Encyclopedia

Python path conflicts cause wrong module versions to be loaded, leading to subtle bugs and errors....

HuggingFace tokenizers library uses Rust backend, which can fail with native errors not visible in Python traceback....

Python multiprocessing fork issues cause CUDA context conflicts and worker process deadlocks....

Python version mismatches between training and inference cause subtle bugs from library incompatibilities....

OpenMPI and general MPI issues cause distributed training to fail at initialization or hang during collectives....

HuggingFace Hub errors occur when downloading models or datasets fails due to auth, network, or rate limits....

SSH key authentication issues prevent passwordless login for distributed training and remote execution....

Conda environment conflicts cause package version mismatches, broken dependencies, and hard-to-debug import errors....

PyTorch CUDA version mismatch with installed CUDA toolkit/driver causes import errors or runtime failures....

Libc (glibc, musl) version mismatches between training and deployment cause binary incompatibility and runtime errors....

Docker image mismatches between dev and production cause code to behave differently due to library versions, CUDA, or system dependencies....

GCC version mismatches cause C++ extension compilation failures and runtime library incompatibilities....

SSL certificate errors prevent downloading models, datasets, or connecting to APIs in ML training pipelines....

Low ulimit values (open files, max processes) cause data loading failures and parallel training issues....

Missing environment variables (CUDA_VISIBLE_DEVICES, HF_TOKEN, MASTER_ADDR) cause silent failures or wrong behavior....

Connection timeouts prevent model downloads, data fetches, and inter-node communication from completing....

Timezone mismatches cause confusion in distributed training logs, scheduled jobs, and time-based events....

Docker containers fail to access GPUs when nvidia-docker or GPU passthrough is not properly configured....

Docker permission errors prevent containers from accessing required resources like GPUs, files, or network....

Python import errors prevent training from starting when modules cannot be loaded....

Python type errors crash training when incompatible types are used in operations....

Python value errors crash training when functions receive values of correct type but inappropriate value....

Python key errors crash training when accessing dictionary keys that don't exist....

Python attribute errors crash training when accessing attributes that don't exist on objects....

Python index errors crash training when accessing list/tuple indices that don't exist....

PyTorch Hub errors prevent loading pretrained models or datasets from torch.hub....

Generic PyTorch CUDA errors indicate various GPU-related issues that prevent training....

Torchvision errors prevent loading pretrained vision models or using vision transforms....

Transformers version mismatches between training and inference cause subtle bugs and performance regressions....

Streaming chat completions from Unsloth Studio's OpenAI-compatible endpoint crash with a RuntimeError about exiting a cancel scope in a different task, on macOS with Python 3.13. The anyio/Starlette s...

Installing a QLoRA/4-bit training stack (Axolotl, TRL, Unsloth) on macOS fails with 'No matching distribution found for bitsandbytes==0.43.0'. bitsandbytes historically shipped only CUDA Linux/Windows...

Importing DeepSpeed fails with NameError/ImportError for _disable_dynamo_if_unsupported because DeepSpeed references a torch internal symbol absent from the installed torch version. Aligning the torch...

Installing DeepSpeed on native Windows fails building the async_io op because it depends on Linux libaio, which has no Windows equivalent. DeepSpeed targets Linux; use WSL2 or a Linux host, or build w...

DeepSpeed import/launch fails with 'cannot import name _get_socket_with_port from torch.distributed.elastic.agent.server.api' because DeepSpeed uses a torch internal symbol that was renamed/removed in...

Importing Unsloth on Windows fails with a UnicodeDecodeError because an internal open() omitted encoding='utf-8' and Windows defaults to cp1252. Upgrade Unsloth (now passes the encoding), or set PYTHO...

A git/editable install of Axolotl reports success, but importing axolotl or running accelerate launch -m axolotl.cli.train fails with ModuleNotFoundError. The src-layout package never landed on the pa...

torch.jit.script on a Megatron @jit_fuser-decorated function fails with 'Unknown type constructor Sequence' on newer torch. TorchScript can't resolve typing.Sequence as used inside the fused function....

Building DeepSpeed's cpu_adam op fails with 'fatal error: cusolverDn.h: No such file or directory' and 'Error building extension cpu_adam'. PyTorch's conda package shipped its own nvcc that shadows th...

Building a DeepSpeed wheel on Windows 11 fails with 'LINK : fatal error LNK1181: cannot open input file aio.lib' and header compile errors. async_io/libaio is Linux-only and some headers don't compile...

Import and environment errors crash training at startup due to missing or mismatched dependencies....

Version mismatches between PyTorch, CUDA, and cuDNN cause cryptic errors....

CUDA driver and runtime version mismatches prevent PyTorch from initializing CUDA....

PyTorch installed without CUDA support prevents GPU training entirely....

The containerized CUDA runtime fails to execute kernels because the host NVIDIA driver is too old to support it....

HPC containers lose GPU access when --nv cannot locate host driver libraries (nonstandard driver install paths, missing nvidia-container-cli, ldconfig cache staleness) or when overlay mounts fail on k...

During init_process_group, PyTorch uses a TCPStore to exchange initial connection information (like ncclUniqueId). The default timeout for TCPStore is often shorter than the NCCL watchdog or the time ...

Poor airflow or a dried-out thermal pad on the HBM stack causes the memory to overheat. High temperatures exponentially increase the rate of transient single-bit flips. The hardware ECC engine correct...

When a PyTorch DataLoader worker exceeds available memory (often due to small Docker `/dev/shm`), it is silently killed by the OS OOM killer. The parent process is waiting on a multiprocessing Queue f...

FlashAttention compiles custom CUDA kernels during installation via `setup.py`. It queries PyTorch for its compiled CUDA version (`torch.version.cuda`) and compares it against the system's `nvcc` comp...

The user installed a pre-built wheel of `flash-attn` compiled against a specific PyTorch and CUDA version (e.g., PyTorch 2.1.0 + cu118), but their runtime environment has a different PyTorch version (...

Building FlashAttention requires compiling massive CUDA templates. If the `ninja` build system is not installed, `setuptools` falls back to standard sequential compilation, which takes hours. Furtherm...

An environment variable forces NCCL to use a static topology XML file that does not match the actual hardware layout of the dynamically allocated compute node....

The FlashAttention kernel hits a hardware capability block (e.g., unsupported SM version) and initiates a fallback path. The fallback Python code attempts to parse metadata that was never instantiated...

Displays as 'Call to open failed: No such file or directory' for /dev/shm. Caused by container environments lacking NUMA support or adequate shared memory limits....

TensorRT requires min to be less than or equal to opt, and opt less than or equal to max, for every dynamic dimension in an optimization profile. The engine build aborts when TensorRT-LLM derives a pr...

FP8 in TensorRT-LLM requires hardware FP8 tensor cores, which exist from compute capability 8.9 onward. A build targeting an Ampere device such as the A100, which reports compute capability 8.0, is re...

When no hardware NVFP4 backend is available, vLLM falls back to an emulation path that unpacks four-bit values through a small constant lookup table. The table is indexed by a tensor that lives on the...

TensorRT-LLM delegates fused attention to FlashInfer, whose FMHA runner is compiled for a specific set of architectures. On a device outside that set the runner raises an unsupported-architecture erro...

The FlashInfer attention backend inside TensorRT-LLM reads fields from an attention metadata object that the release bundled beside it does not define. Serving fails with a plain Python AttributeError...

NGC containers ship a set of extensions compiled against a specific NumPy major version. Installing anything that pulls a newer NumPy replaces the runtime those extensions were built for, and the next...

An interpreter upgrade changes what pickle will serialise, and traceback objects stop qualifying. Any machinery that ships an exception between ranks then fails while packing it, so the error the clus...

Batch size changed between gradient accumulation micro-steps. Accumulation sums gradients across micro-batches before one optimizer step, so a varying batch silently reweights the average. Accelerate ...

The AWQ GEMM kernel was launched with a grid or block configuration the device rejects. AWQ kernels assume shapes divisible by their tile size; an unusual hidden dimension or a batch of 1 can produce ...

FlashAttention produced non-finite values under FP16 autocast. Overflow can originate in the inputs, attention logits, scaling, or an unsupported kernel path, so the first non-finite tensor determines...

The ExLlama kernel cannot serve this GPTQ checkpoint: act-order (desc_act=True) combined with a grouped quantization is unsupported by that kernel path. The model is fine; the kernel choice is not. Th...

bitsandbytes loaded its CPU-only build, or the CUDA build it selected does not match the installed toolkit or the GPU architecture. Quantized (8-bit / 4-bit) paths silently fall back or fail outright....

The installed bitsandbytes CUDA binary does not contain a 4-bit kernel for the GPU compute capability in use. This is an architecture or binary compatibility failure, not a model-memory shortage. This...

Attempting to perform CPU synchronization or stream switching (e.g..item() or dynamic tensor operations) while a CUDA graph is actively capturing operations. This entry explains how to confirm the cau...

A CUDA graph was replayed after the memory it captured was freed or reallocated. Graphs record raw device pointers, so any tensor whose storage moved between capture and replay leaves the graph readin...

A kernel executed an instruction the GPU does not implement. Almost always an architecture mismatch: the binary contains SASS for a different compute capability, or was JIT-compiled from PTX targeting...

cuDNN convolution operation failed, unsupported algorithm, channel layout mismatch, or GPU compute capability limit. This entry explains how to confirm the cause, apply the fix, and separate it from a...

A native custom CUDA extension crashed the process during backward execution. The fault is inside the extension or its ABI boundary, not an NCCL collective simply because distributed ranks exit afterw...

AutoencoderKL ran out of VRAM while decoding tiled latent intermediates. Tiling reduces the peak for the main VAE operation, but overlap buffers, output assembly, dtype, and concurrent model residency...

The container runtime could not select or initialize the NVIDIA GPU runtime. The host toolkit, runtime configuration, driver visibility, or requested GPU capability is missing before the application s...

Enroot exhausted the user or filesystem quota while importing and extracting the SquashFS image. Free space elsewhere on the node does not help when the configured cache or data path has a separate qu...

llama.cpp/GGUF load failure, file is not GGUF (old GGML), GGUF version newer than the runtime, truncated download, or quantization type unsupported by this build. This entry explains how to confirm th...

The Rust tokenizer's thread pool was already active when the process forked, so the child inherits locks held by threads that do not exist in it. The warning precedes either a silent hang in the DataL...

TorchInductor exhausted a private CUDA Graph memory pool even though total free memory may appear sufficient. Captured graphs retain stable addresses, so inactive blocks in the private pool cannot alw...

The NVIDIA device plugin could not load libnvidia-ml.so, so it never registered nvidia.com/gpu with the kubelet. Every GPU pod on the node stays Pending with "Insufficient nvidia.com/gpu" while the no...

The GPU's hardware video decoder ran out of concurrent sessions. NVDEC session count is a fixed hardware/driver limit, and consumer boards are capped far below data-center parts regardless of availabl...

The container runtime could not find the NVIDIA device nodes on the host. The kernel modules are not loaded, or the node was created after the container runtime cached its view, the GPU is invisible i...

Two OpenMP runtimes were loaded into one process (Intel/LLVM libomp alongside GNU libgomp). Behaviour is undefined, in practice thread pools collide, causing deadlocks, oversubscription, or crashes in...

A LoRA adapter was attached to a quantized (int8/int4) base layer and the training path tried to make the quantized weight itself require gradients. Only the floating-point adapter weights are trainab...

A ROS 2 publisher overwrote shared-memory ring buffer entries a subscriber had not yet consumed. The consumer is slower than the producer, so frames are being lost, and with zero-copy transport the su...

safetensors header/metadata error, file truncated (interrupted download/save), wrong format (HTML error page saved as .safetensors), or reader raced a writer. This entry explains how to confirm the ca...

TorchDynamo captured the graph, but TorchInductor failed while lowering an operation to generated code. The useful cause is the nested LoweringException and its target operator, not the outer BackendC...

PyTorch: TorchDynamo encountered a graph break due to unsupported Python construct. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent torch-compile failures....

The INT8 observer saw NaN or Inf activations while calibrating. Quantization scales are derived from observed ranges, so a single non-finite activation poisons the scale for that tensor and every valu...

NVIDIA PTX assembler failed on Triton output. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent torch-compile failures....

Triton kernel requested too much shared memory. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent quant failures....

GPU memory clock throttling reduces memory bandwidth and training performance when memory is under-utilized....

NVLink errors cause intra-node GPU communication failures and degraded multi-GPU performance....

GPU thermal throttling reduces clock speeds to prevent overheating, causing training to be slower than expected....

GPU power cap limits GPU power consumption, reducing performance for power-constrained deployments....

CUDA driver crash terminates all GPU processes on the node, losing unsaved training progress....

GPU overheating causes thermal throttling, performance degradation, and potential hardware damage....

GPU not detected errors prevent training from starting when the system cannot see the GPU....

CUDA illegal instruction errors crash training when the GPU encounters unsupported instructions, often from binary mismatches....

GPU clock throttle reduces performance when power or thermal limits are reached....

GPU memory bus errors corrupt data transfer between GPU cores and memory, causing silent data corruption or crashes....

GPU MIG mode splits a GPU into multiple instances, which can cause issues with PyTorch and NCCL if not configured correctly....

High GPU ECC error rates indicate degrading memory hardware. Correctable errors accumulating signal impending failure....

GPU memory bandwidth limits reduce training performance when compute exceeds memory access speed....

NVLS collective failures on Hopper/NVLINK4 nodes cascade from Fabric Manager restarts, producing Xid 31 MMU page faults and illegal memory accesses across the node....

P2P fabric-handle import between containers on a multi-node NVLink (MNNVL) GPU fails with CUDA 'invalid device ordinal' due to a CUDA driver bug, not NCCL....

NVLink bandwidth degradation can be a single failing link or a whole-node NVSwitch fault. Comparing per-GPU NVLink bandwidth across all 8 GPUs tells them apart....

The GPU System Processor (GSP) stops responding to RPCs, logged as Xid 119. The GPU becomes unresponsive and usually needs a reset....

A GPU drops off the PCIe bus (Xid 79) and disappears from the system. A critical hardware/connectivity fault that ends training on that node until the GPU is recovered....

The GPU remaps a failing memory row after ECC errors. A recovery mechanism, but a rising remap count signals degrading memory that may need RMA....

Xid 48 is NVIDIA's code for uncorrectable GPU memory errors. It crashes training and corrupts model weights. Denpex correlates Xid 48 events with NCCL failures to identify the originating GPU and trig...

Xid 79 means the GPU lost PCIe connectivity entirely. Training dies immediately. Denpex identifies which GPU, node, and the chain of events that led to the GPU fall. Root caused by bad PCIe, power eve...

CUDA device assertion failures indicate GPU hardware problems that crash training with ambiguous error messages. Denpex maps the device assertion to the specific GPU and operation, distinguishing hard...

KV cache corruption or NaNs in key_value tensors occurring during decoding on hardware topologies that do not fully support PCIe P2P memory sharing....

The display driver crashes and recovers (or fails to recover) due to a long-running CUDA kernel triggering the OS watchdog....

On Intel data-center GPUs, engine hangs surface as i915 "GPU HANG"/engine-reset dmesg lines and GuC (scheduling microcontroller) load or crash messages. Compute contexts die on reset; repeated hangs i...

NVLS (NVLink SHARP) offloads collective operations to the NVSwitch hardware. An incompatibility between the NCCL library version, the CUDA toolkit, and the NVIDIA display driver can cause incorrect me...

PCIe Access Control Services (ACS) is enabled in the host motherboard's BIOS or the Linux kernel. ACS forces all PCIe peer-to-peer traffic to be routed up to the Root Complex (CPU) for IOMMU translati...

One specific worker experiences a CUDA OOM (e.g., due to memory fragmentation or an unusually large sample) and crashes. PyTorch's default process group error handling doesn't forcefully abort the rem...

During heavy compute operations, modern GPUs (like the A100 or H100) experience massive microsecond-level power spikes. If the Power Supply Unit (PSU) or the 12V PCIe power cables cannot handle the tr...

The physical VRAM (HBM or GDDR) on the GPU has degraded over time due to thermal stress or manufacturing defects. A double-bit error is mathematically uncorrectable by the GPU ECC engine, meaning data...

Poor physical seating of the GPU in the PCIe slot, or a degraded PCIe riser cable, degrades signal integrity. When the system shifts into high-bandwidth PCIe Gen4/Gen5 states for inter-GPU communicati...

The GPU's internal processors (like the Power Management Unit or GSP) encountered a diagnostic event, such as a thermal anomaly, voltage droop, or firmware bug, and halted execution to prevent corrupt...

The GPU engine received an illegal command or entered an invalid state, commonly triggered by thermal throttling, power instability, or severe CUDA context corruption from conflicting workloads....

A physical hardware fault on the NVLink connection between GPUs. This can be caused by signal integrity degradation, mechanical stress, or connector contamination....

The CPU lost communication with the GPU over the PCIe bus. Common causes include inadequate power supply under high load, a loose PCIe connection, faulty riser cables, or active state power management...

The GPU hardware detected an uncorrectable memory error in VRAM. However, it successfully 'contained' the error to the specific application using that memory page, preventing a full system crash....

A multi-bit memory error occurred in the High Bandwidth Memory (HBM) that the hardware ECC engine cannot correct. To prevent silent data corruption, the GPU driver marks this as a 'sticky' hardware er...

The HBM is rapidly degrading, triggering single-bit ECC errors at an extremely high frequency. The GPU's self-healing mechanism dynamically retires and remaps these bad memory pages (Xid 63). Doing th...

A transient hardware fault occurred in an unprotected part of the GPU memory pipeline (e.g., a multi-bit flip that bypassed ECC, or a fault in the L1 cache/SRAM). This resulted in an incorrect matrix ...

The GPU detected a bad HBM memory page and attempted to record it in the InfoROM (EEPROM) for permanent retirement. However, the InfoROM is either full or physically degraded, preventing the page rema...

Marginal signal integrity on the PCIe riser cable or motherboard slot under heavy thermal load causes a Malformed Transaction Layer Packet (TLP). The PCIe root complex detects the fatal error via AER ...

Thermal expansion or microscopic dust in the PCIe slot causes transient connection issues during Active State Power Management (ASPM) or link retraining. The PCIe controller automatically negotiates a...

When `pin_memory=True`, PyTorch spawns a background thread to copy pageable host memory to pinned (page-locked) host memory. If the objects returned by the Dataset are not standard tensors (e.g. custo...

The Kubernetes scheduler is topology-blind. It assigns the Pod to a node based on total available capacity. However, the Kubelet's Topology Manager on the node (often configured with the 'single-numa-...

The NVIDIA device plugin relies on the NVML library to discover GPUs and report their health to the kubelet. If the underlying GPU hardware experiences a fault (e.g., an Xid error causing it to fall o...

FlashAttention v2 heavily optimizes memory access and matrix multiplication using specific hardware features introduced in NVIDIA's Ampere architecture (Compute Capability 8.0+), such as asynchronous ...

A brief hardware network event (link flap, switch reset, optical transceiver issue) breaks the reliable InfiniBand (RC) connection between nodes. NCCL encounters an unrecoverable RDMA error and raises...

A severe hardware fault (often power delivery or PCIe signal integrity) causes the GPU to disconnect from the PCIe bus entirely. The CUDA runtime loses contact with the device, and NCCL operations imm...

Physical silicon degradation in the High Bandwidth Memory (HBM) modules causing multi-bit data corruption in a single memory word....

A transient transmission failure on the NVLink or RoCE fabric propagates corrupted state to the SM. The GPU isolates the SM fault to prevent data corruption, emitting a containment Xid without an actu...

The GPU System Processor (GSP) firmware crashes internally, terminating the PCIe link negotiation. This can occur even under near-zero utilization or idle states....

Highly synchronous computational workloads (like dense GEMMs in LLMs) cause periodic power oscillations that resonate with the physical PDN, causing voltage droops that violate logic gate timing margi...

Often occurs randomly when the GPU utilizes aggressive power saving and attempts a rapid transition from its lowest state (PCIe Gen 1) to a high-performance state (Gen 3/4)....

Caused by aggressive CPU contention (like massive dataloader thread counts) starving the NVIDIA kernel module of the CPU cycles required to acknowledge GPU interrupts....

The Neuron runtime's DMA ring buffer filled, transfers were queued faster than the device drained them. The accelerator is keeping up with compute but not with the data being pushed at it. This entry ...

The Neuron compiler exceeded its time budget compiling the graph. Trainium compiles the whole model graph ahead of execution, and compile time grows sharply with graph size and with the number of dist...

A Blackwell B200 module detected an unsafe temperature difference between its two dies and entered emergency throttling. The delta points to uneven cooling or contact, not ordinary workload-wide therm...

An uncorrectable GPU memory error means ECC could not repair the affected data. On modern NVIDIA data-center GPUs, the next action depends on whether the error was contained, the recovery-action flag,...

Liquid cooling flow dropped below the safe threshold. This is a facilities fault with a very short fuse, direct-liquid-cooled GPUs have little thermal mass, so an emergency power-off follows quickly a...

DCGM observed an increase in NVLink CRC flit errors and marked the link degraded. A rising counter indicates signal or link integrity trouble, not a collective software timeout. This entry explains ho...

HBM row remapping failed: the GPU has exhausted its reserve of spare memory rows and can no longer retire failing ones. This is a terminal hardware state, the GPU will keep producing uncorrectable ECC...

The coherent NVLink-C2C path between the Grace CPU and Hopper GPU reported a fatal link or coherency failure. Continuing on the node risks invalid data movement, so this is a platform health event rat...

A TPU ICI link at the reported mesh coordinate is down, so the accelerator slice cannot provide the topology expected by the distributed program. This is a slice or platform health fault, not an XLA g...

A Habana collective timed out waiting for a peer. Like every collective timeout this names the ranks that WERE waiting, not the one that failed to arrive, the OAM module in the message is the reporter...

NVIDIA Fabric Manager failed to initialize, so NVSwitch routing was never programmed. On an HGX/DGX baseboard every NVLink-based collective is unavailable until this is fixed, jobs either fall back to...

GPU is thermal-throttling, clocks reduced to prevent damage, causing silent performance degradation. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent telemetr...

Xid 119/120 - GSP (GPU System Processor) RPC timeout. The driver-firmware RPC channel deadlocked; the GPU resets and the CUDA context is lost. Known driver/GSP-firmware combo bugs are the usual cause ...

GPU MMU page fault (Xid 31), often an NVLS/Fabric Manager cascade. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent xid failures....

Xid 45 records driver cleanup after a CUDA context was terminated. It is usually evidence of an earlier application exit, GPU reset, or administrator action, not the initiating hardware fault. This en...

Xid 62, internal microcode error. The GPU firmware hit an unrecoverable internal fault. The device usually survives a reset, but a repeat on the same serial is a hardware or firmware defect rather tha...

Xid 79 means the driver can no longer reach the GPU over PCIe. The device stopped responding to config-space reads mid-operation, so the kernel dropped it. Every process holding a CUDA context on that...

GPU ECC fault (Xid 48/94/95, double-bit/uncorrectable ECC, or row-remap event) caused the CUDA/runtime failure. The CUDA error is downstream hardware fallout, not the root cause. This entry explains h...

An NVLink switch tray port failed link training, the physical layer could not establish a clean signal. On an NVL72-class rack this removes a path from the fabric, and collectives crossing it either f...

NVLink Correctable/Recovery Errors - The link is experiencing high noise, causing CRC failures and replays. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent n...

SXM power brake event. GPU exceeded power budget; system forced power limiting. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent telemetry failures....

PCIe link speed or width downgraded, the GPU is not running at Gen5 x16 (or expected width). This entry explains how to confirm the cause, apply the fix, and separate it from adjacent telemetry failur...

RCCL could not map a peer IPC handle across AMD Infinity Fabric, so the ring could not be formed. The AMD equivalent of an NCCL P2P transport failure, peer access between GPUs is unavailable or blocke...

HIP runtime returned Out of Memory error. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent non-nvidia failures....

Activation memory spikes during specific operations cause transient OOM even when steady-state memory fits....

torch.cuda.memory_summary() helps debug memory issues but output can be overwhelming without understanding it....

Debugging PyTorch's caching allocator helps identify memory issues but requires understanding its behavior....

CPU RAM exhaustion during training kills the process or causes swapping that drastically slows training....

Gradient accumulation can cause unexpected memory spikes when scaled batch size is large or loss is not properly reduced....

Memory leaks in DataLoader workers cause steadily growing memory usage over training epochs....

CUDA Graphs capture memory allocations that are hard to release, causing memory leaks across graph replays....

KV cache memory grows with sequence length in transformer inference and some training setups, causing OOM at long contexts....

Inference memory leaks cause growing GPU memory usage during serving, eventually OOM-ing the server....

Pinned (page-locked) memory overuse causes system RAM exhaustion and slow CPU-GPU transfers when over-allocated....

Activation distillation memory costs grow with teacher model size and student hidden dimension matching requirements....

Flash Attention saves memory by not materializing the full attention matrix, but has shape and dtype constraints....

Transformer cache (KV cache, past_key_values) memory grows with sequence length and can cause OOM at inference....

Dataset cache memory usage grows when transformations or augmentations are applied before batching....

Gradient checkpointing trades compute for memory; misconfiguration can either not save memory or drastically slow training....

torch.compile can use additional memory for graph compilation, guard evaluation, and dynamic shape handling....

cuDNN benchmark mode can use more memory for algorithm search, and not all algorithms work with all configs....

Tensor parallel training splits model across GPUs, but communication buffers and synchronization can still cause OOM....

Loading checkpoints can temporarily double memory usage because both old and new model states exist in memory....

Safetensors load errors occur from corrupted files, version mismatches, or architecture mismatches with the loading model....

Tensor views (slices, reshapes, transposes) can hold references to large base tensors, preventing memory release....

Pipeline parallel memory issues arise from stage imbalance, bubble overhead, and activation storage across micro-batches....

DeepSpeed OOM errors occur when ZeRO partitioning, CPU offload, or activation partitioning is misconfigured....

FSDP all-gather operations can timeout when parameters are large or network is slow, causing training to fail....

CUDA context leaks occur when CUDA contexts are not properly destroyed, accumulating GPU memory across processes....

CUDA caching allocator fragmentation causes OOM despite enough total free memory, due to non-contiguous blocks....

Optimizer state memory (Adam: 2x model size, AdamW: 2x) can exceed model size memory, dominating total usage....

Transformer attention memory grows quadratically with sequence length; long-context training requires Flash Attention or similar....

Embedding layer memory grows with vocabulary size and can dominate total memory for large vocab models....

CUDA out of memory is the most common training error, occurring when GPU memory is exhausted....

Activation checkpointing compatibility issues arise when using torch.compile, FSDP, or DDP with checkpointing....

Python OOM kills the training process when Python's heap memory is exhausted, even if GPU memory is fine....

CUDA shared memory limits are reached when kernels use too much shared memory per block....

CUDA caching allocator issues cause memory not being released back to the GPU when expected....

CUDA Unified Memory (UVM) errors cause page faults and performance issues when memory oversubscription occurs....

Activation checkpointing misuse causes either OOM (not enough checkpointing) or slow training (too much checkpointing)....

ZeRO CPU/NVMe offload materializes parameters and optimizer states in host RAM on every rank, exhausting system memory and getting the process OOM-killed with no Python traceback....

DeepSpeed FusedAdam optimizer triggers CUDA illegal memory access errors on NVIDIA H100 GPUs. This is a compatibility issue between FusedAdam's CUDA kernels and H100's SM90 architecture. Denpex detect...

GPU memory is not released after trainer.train()/engine teardown, so repeated training calls (CV folds, hyperparameter sweeps) leak until OOM. DeepSpeed left gradient-accumulation/backward hooks attac...

Fine-tuning a large model (e.g. Flan-T5-XXL 11B) with ZeRO-3 uses far more GPU memory than expected or OOMs at load because the model was constructed outside DeepSpeed's zero.Init context. Every rank ...

CUDA memory leaks accumulate over training, eventually causing OOM. Denpex tracks allocation patterns to identify leaks....

Memory fragmentation causes OOM even when total free memory is sufficient, because no contiguous block is available for the allocation....

Host OOM (out of memory) kills training processes when system RAM is exhausted, often silently....

CUDA memory allocation fails when the requested memory block cannot be allocated, often due to fragmentation or insufficient total memory....

The kernel oom-killer (or cgroup limit) kills python workers whose RSS grew unbounded. Shard caches, tokenized-in-RAM datasets, prefetch queues. Surface strings ("Out of memory", "Killed") route engin...

A single rank hits a CUDA Out of Memory error during the forward or backward pass and crashes or raises an exception. Because the process group isn't cleanly torn down, the surviving ranks continue to...

When using `FullStateDictConfig(offload_to_cpu=True)` without setting `rank0_only=True`, FSDP instructs every single rank to independently gather and reconstruct the full model parameters into its own...

FSDP performs frequent allocations and deallocations of varying sizes (flattening, padding, un-sharding, sharding). This usage pattern severely fragments the CUDA caching allocator. Memory gets trappe...

By default, DeepSpeed's ZeRO-3 config parameter `gather_16bit_weights_on_model_save` is often set to true. This instructs DeepSpeed to unshard and consolidate all model weights onto a single GPU (Rank...

When using ZeRO-Offload (CPU offloading for parameters/optimizer states), DeepSpeed enables `pin_memory: true` by default. Pinned memory (page-locked memory) cannot be swapped to disk. Offloading larg...

DeepSpeed ZeRO-3 uses a `prefetch_bucket_size` (defaulting to a very large number like 50M) to preemptively AllGather upcoming layers into VRAM to hide communication latency. For large models or restr...

Xid 31 occurs when the GPU attempts to access a virtual address that is not mapped in the GPU page tables. While this can be a kernel bug, it frequently happens under severe memory pressure or thrashi...

The PyTorch caching allocator reserves large blocks of memory and splits them for individual tensor allocations. Over time, memory becomes fragmented into many small blocks, preventing the allocation ...

The developer accumulates the loss tensor across batches for logging (e.g., `total_loss += loss`). Because `loss` is a PyTorch tensor attached to the computation graph, accumulating it keeps the entir...

The evaluation loop is executing without `with torch.no_grad():` or `@torch.inference_mode()`. Therefore, PyTorch continues to build the autograd computation graph during evaluation, storing activatio...

Optimizers like Adam and AdamW maintain moment states (running averages of gradients and squared gradients) for every parameter. For a model with N parameters, the optimizer state requires 2 * N memor...

Standard multi-head dot-product attention computes an attention matrix of size (batch_size, num_heads, seq_length, seq_length). The memory complexity is O(N^2) with respect to the sequence length. As ...

An index passed to a PyTorch `nn.Embedding` layer exceeds the configured `num_embeddings` (e.g., passing token ID 1000 to an embedding matrix of size 1000, where max valid is 999). Because CUDA execut...

When `torch.multinomial` receives a probability distribution tensor containing `NaN` values, negative values, or all zeros, the CUDA kernel fails to build a valid Cumulative Distribution Function (CDF...

ZeRO-3 partitions model parameters and offloads them to CPU memory. When a custom operation (like a custom regularizer or un-traced optimizer step) tries to access a parameter tensor directly, it pass...

When using variable length sequences in FlashAttention (e.g., passing `cu_seqlens`), if the actual sequence length in the data exceeds the `max_seqlen_k` or `max_seqlen_q` parameter passed to the kern...

A CUDA Out Of Memory (OOM) error is an application-level exception thrown by the CUDA runtime to the framework (e.g., PyTorch). Unlike a system RAM OOM where the Linux kernel's OOM-killer sends a SIGK...

Kubernetes GPU time-slicing multiplexes compute execution time, but it provides absolutely zero hardware-level memory isolation. If a physical GPU with 24GB VRAM is time-sliced into 4 logical GPUs, Ku...

vLLM aborts during engine initialisation because, after loading model weights and profiling the activation peak, nothing is left inside its memory budget to allocate a single paged KV cache block. The...

vLLM returns an error for a request whose prompt tokens plus requested completion tokens exceed the context window the server was started with. The served window is not always the one advertised on th...

Raising the number of tokens vLLM may batch per step improves time to first token and throughput, and it also raises the peak activation memory a prefill step needs. When that peak collides with the m...

CPU offloading trades GPU memory for PCIe bandwidth. Parameters, gradients or optimizer state cross the bus every step, and when the transfer cannot hide behind compute the step becomes bound by the l...

A large pinned host-memory allocation exceeds the host pinning path; the generic CUDA debugging footer is not an observed device assertion. This entry explains how to confirm the cause, apply the fix,...

CUDA could not export an interprocess memory handle for the allocation. Device memory pressure, an ineligible allocation, or exhausted IPC and driver bookkeeping can cause this before the receiving pr...

FAISS could not allocate GPU memory for the index. A flat GPU index holds every vector in VRAM, so the requirement is vectors × dimensions × 4 bytes plus working space, it scales linearly and unforgiv...

Milvus could not load a vector index segment into GPU memory. Segments are loaded whole, so a single oversized segment fails even when total free VRAM across the collection looks sufficient. This entr...

Sentence Transformers exceeded VRAM during encoding because the token count, batch size, output handling, and other GPU residents exceeded the model memory budget. Sequence length can dominate even wh...

PyTorch multiprocessing could not map another shared-memory region because /dev/shm is full or capped too low. This consumes shared memory, not filesystem disk and not GPU VRAM. This entry explains ho...

Managed-memory pages are bouncing between host RAM and GPU HBM faster than either processor can reuse them. The migration traffic saturates the interconnect and can surface as an Xid 31 page fault or ...

NCCL CUDA failures occur when the underlying CUDA runtime has issues, causing all NCCL operations to fail....

NCCL P2P (peer-to-peer) communication can be disabled, forcing fallback to slower transport....

NCCL hangs can be hard to detect because they don't produce errors until the watchdog timeout. Proper monitoring helps catch them early....

Wrong rank configuration in NCCL causes collectives to fail or produce incorrect results when rank assignments don't match expectations....

NCCL peer-to-peer (P2P) communication issues cause pipeline parallelism and tensor parallelism to fail or be slow....

Gloo backend issues occur when using Gloo instead of NCCL for distributed training, especially on CPU or limited GPU setups....

TCP port exhaustion occurs in distributed training when many connections exhaust ephemeral port range....

TorchElastic errors occur when training jobs need to be elastic (resize dynamically) but configuration is wrong....

Horovod setup errors occur when MPI or NCCL integration is misconfigured, preventing elastic or multi-GPU training....

NCCL InfiniBand HCA (Host Channel Adapter) configuration mismatches cause slow or failed inter-node communication....

NCCL IPv6 issues occur when NCCL tries to use IPv6 but the network only supports IPv4, or vice versa....

Pipeline parallelism has idle time (bubble) at start and end of each pipeline, reducing efficiency for small models or small micro-batches....

AWS EFA (Elastic Fabric Adapter) driver issues cause NCCL to fall back to slower TCP/IP for inter-node communication....

DDP port conflicts prevent distributed training from starting when the master port is already in use....

RDMA configuration issues prevent high-bandwidth, low-latency inter-node communication in GPU clusters....

NCCL rank failures occur when one or more ranks fail to initialize or join the distributed group....

NCCL version mismatches between nodes cause collective operations to fail or hang in distributed training....

DDP setup errors occur when distributed data parallel training is not properly configured, preventing multi-GPU training....

NCCL bucket size mismatches in DDP cause inefficient gradient reduction, hurting performance....

All-reduce deadlocks occur when DDP/FSDP all-reduce operations are mismatched across ranks or with batch norm....

NCCL error 2 is an internal assertion failure in NCCL itself, often from corrupted state or hardware issues. Denpex correlates with hardware diagnostics....

NCCL initialization times out when nodes cannot establish the communication channel within the allowed time....

NCCL RDMA performance degrades when GPUDirect RDMA is unavailable or misconfigured, causing slow multi-node training....

NCCL asynchronous error handling issues cause silent failures or deadlocks when collective errors are not properly handled....

NCCL GPUDirect over RoCE (RDMA over Converged Ethernet) requires proper configuration for high performance....

NCCL point-to-point bandwidth issues cause slow intra-node GPU communication when P2P is disabled or limited....

NCCL exhausts the NVSwitch's fixed pool of NVLink SHARP (NVLS) multicast slots, then treats the binding failure as fatal instead of falling back to a non-NVLS transport....

NCCL queue-pair creation times out specifically during distributed checkpoint save at large scale on RoCE, while normal training collectives succeed....

A generic CUDA 'unknown error' (999) surfaces during NCCL communicator initialization, usually pointing to a GPU in a bad state or a driver/runtime mismatch rather than an NCCL bug....

DataParallel model replication fails at the SharedInit step with a CUDA invalid-argument error, often tied to legacy nn.DataParallel inside complex launchers....

FSDP all-gather fails with ncclSystemError on one node during collective init over InfiniBand, while a TCP fallback succeeds. Pointing to a per-node fabric or plugin issue....

Querying the NCCL RAS subsystem during training on NCCL 2.27.3 returns corrupted communicator data and can crash the job via TCPStore connection failures....

Intermittent NCCL internal errors during init on partial-node allocations, caused by the NIC-fusion feature remapping NET IDs so the hardcoded fallback NET/0 no longer exists....

NCCL fails to read the RoCE GID on containerized/macvlan setups because GID iteration starts at index 0 where leading entries are all-zero, returning EINVAL....

A dangling group-job pointer causes a segfault during ncclCommSplit when using threads with non-blocking communicator init; fixed in NCCL 2.26.2....

A race in the NCCL RAS subsystem segfaults during init when a RAS command runs before peer info is populated; fixed in NCCL 2.27.3....

NVLS did not support dual-port NIC transmission, producing duplicate head-rank entries in the proxy loop and heap corruption; fixed in NCCL 2.19.4....

Inconsistent NIC enumeration (e.g. mlx5_3 before mlx5_0) on one node breaks NCCL topology detection and causes random multi-node segfaults; fixed in NCCL 2.23.4....

A stale socket file descriptor is reused on retry after a failed connect, triggering ECONNABORTED during NCCL communicator init; fixed in NCCL 2.24....

NCCL P2P topology detection lacks SM120 (Blackwell) support, using a wrong shared-memory maximum so peer-to-peer connections fail on RTX 5090....

On GB200 NVL72, NCCL's recursive MNNVL topology search overruns the stack at world_size >= 44, especially with ulimit -s unlimited; fixed in NCCL 2.28....

NCCL's proxy thread tries to connect over IPv6 while the peer only answers on IPv4, producing 'Proxy Connect failed' on dual-stack hosts....

NCCL's NIC selection/path-finding can deadlock when a node has an odd number of HCAs (specifically three), while 1, 2, or 4+ work fine....

RoCE MTU mismatches between the NIC and switch cause NCCL RDMA queue-pair errors (completion error 12, vendor error 129). Training hangs or crashes during NCCL init or the first collective. Denpex det...

NCCL mismatched collective errors occur when different ranks call different collective operations or call the same collective with different parameters. This is a programming error, not a hardware fau...

NCCL's OFI (OpenFabrics Interface) transport fails to register memory for RDMA when the OS memlock limit is too low. Training crashes at NCCL init with 'Unable to register memory RC:12'. Denpex detect...

When Access Control Services (ACS) is disabled in the BIOS, GPU Direct RDMA (GDR) fails with RDMA completion error 4 and vendor error 81. NCCL performance degrades severely or training crashes. Denpex...

NCCL topology detection fails when the NIC's PCI bus ID is missing from the system topology XML. This causes NCCL to fall back to a suboptimal transport or fail to use GPU Direct RDMA. Denpex detects ...

Setting NCCL_BUFFSIZE too large causes NCCL Socket transport to stall or hang during communication. This is a configuration error where users increase the buffer size hoping for better performance but...

NCCL v2.18.3 introduced a topology detection regression that misidentifies GPU-NIC affinity on some systems, causing NCCL to use a suboptimal communication path. Denpex detects the regression from NCC...

NCCL allreduce hangs stall training when collective operations can't complete. Denpex identifies the stuck rank or rank pair....

NCCL communication timeouts abort training when collective operations take too long to complete....

NCCL initialization errors prevent distributed training from starting due to network or configuration issues....

NCCL rank stuck errors occur when one rank cannot keep up with the collective operation, slowing down all ranks....

NCCL version conflicts prevent proper collective operations when nodes have different versions installed....

When cluster DNS degrades, every layer above it fails with ITS own vocabulary: gaierror in python, client-socket failures in torch.distributed, bootstrap failures in NCCL. Teams debug the fabric while...

vLLM distributes each step to its workers through a shared-memory ring buffer. When a worker fails to publish within the read deadline, the reader raises a bare TimeoutError from acquire_read, the exe...

Expert-parallel routing sends every token to the ranks holding its chosen experts and gathers the results back, an all-to-all exchange whose volume depends on what the router chose. On large NVLink do...

GDRCopy could not pin the GPU virtual address into the CPU-visible BAR1 aperture. The allocation is not eligible for peer-direct mapping, BAR1 space is exhausted, or the kernel module and driver disag...

Gloo TCP store connection failed (connection refused or DNS resolution failed), ranks cannot reach the rendezvous point. This entry explains how to confirm the cause, apply the fix, and separate it fr...

An RDMA queue pair could not transition to Ready-To-Receive, so the connection never established. On RoCE this is nearly always a GID index or MTU mismatch between the two endpoints; on InfiniBand it ...

The NCCL tree algorithm stalled on this topology. Tree is chosen automatically for some rank counts and message sizes, and an irregular or asymmetric rank layout can leave it waiting on a peer that ne...

NCCL could not allocate the contiguous pinned host buffer requested by NCCL_BUFFSIZE. The configured buffer and communicator count exceeded available lockable memory or the process memlock limit. This...

Traffic spread across multiple NICs arrived out of order. RoCE is highly sensitive to reordering, go-back-N retransmission means a reordered flow collapses throughput rather than degrading gracefully,...

NCCL could not bring up a transport (IB/EFA/NET) between two ranks. "Resource temporarily unavailable" here usually means the RDMA device ran out of queue pairs or memory registrations, or the contain...

GPUDirect RDMA was attempted between a GPU and a NIC on different NUMA nodes. The transfer has to cross the CPU interconnect (UPI/Infinity Fabric), which is far slower than a local path and on some pl...

This NCCL warning occurs while the NVLink SHARP transport is creating or registering multicast resources. The line identifies the failing NVLS path, but it does not prove whether the owner is applicat...

An NCCL ring peer attempted a TCP connection before the expected listener was reachable. The destination rank may have crashed, advertised the wrong interface, or been blocked by host or network polic...

NCCL tried to reach a peer on an interface that cannot route to it. Almost always NCCL_SOCKET_IFNAME selecting the wrong NIC, a docker0/lo/management interface instead of the fabric, so ranks advertis...

An NCCL bootstrap TCP connection was accepted and then reset by a peer before communicator setup completed. The resetting rank may have exited, selected a different interface, or lost network reachabi...

nvidia-peermem could not create another peer-direct mapping because the GPU BAR1 aperture has no usable space for it. Stale peer mappings, an undersized BAR1 window, or an unsupported platform layout ...

NVLS initialization failures occur while NCCL is preparing NVSwitch multicast resources for collective offload. Diagnose support, allocation, software state, and fabric health separately instead of tr...

InfiniBand Subnet Manager Flap - The fabric routing changed during the run, dropping active connections. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent infi...

Priority Flow Control pause frames saturated a link. PFC exists to make RoCE lossless, but a congested or misconfigured fabric can propagate pause backward through switches, a "congestion tree" that s...

NaN detection and skipping prevents corrupted gradients from propagating but requires careful implementation....

Training without learning rate warmup or decay causes slow convergence and poor final performance....

Misusing the results of a learning rate finder causes poor training setup choices....

Gradient accumulation with BatchNorm causes incorrect normalization because BN computes statistics on smaller sub-batches....

Label noise in training data causes poor model generalization and unexpected training behavior....

Spectral norm clipping for GAN training has different requirements than gradient clipping for other models....

Weight decay misconfiguration causes overfitting, underfitting, or training instability depending on the wrong choice....

Exponential Moving Average (EMA) decay misconfiguration causes poor model averaging and training instability....

Lookahead optimizer issues arise from improper k (inner steps) or alpha (slow weight) values that destabilize training....

Ranger optimizer combines RAdam and Lookahead; misconfiguration can cause training instability or poor convergence....

Adafactor optimizer issues arise from incorrect epsilon, scaling factor, or relative step handling....

SAM optimizer issues arise from the two-forward-pass requirement, learning rate issues, or rho misconfiguration....

Lion optimizer issues arise from incorrect learning rate (typically 3-10x lower than Adam) or momentum configuration....

BF16 and FP16 mixed precision have different numerical properties; choosing the wrong one causes instability or wasted memory....

Spectral normalization can cause mode collapse or training failure when applied incorrectly to GANs....

SWA training issues arise from incorrect averaging frequency, learning rate schedule for SWA, or BN update steps....

Polyak averaging issues arise from incorrect averaging window, frequency, or weight initialization....

Curriculum learning issues arise from poorly designed difficulty progression that hurts rather than helps training....

Cyclic LR scheduler issues arise from incorrect base/max LR, step size, or mode (triangular, triangular2, exp_range)....

One-cycle policy issues arise from incorrect max_lr, momentum range, or training duration that destabilize training....

Applying weight decay to LayerNorm and bias parameters hurts training and can prevent convergence in transformers....

Fine-tuning failures occur when pretrained model knowledge is destroyed by aggressive learning rates or insufficient data....

Loss curve anomalies (spikes, plateaus, oscillations) indicate underlying training issues that need diagnosis....

Missing warmup causes early training instability, especially with large learning rates or transformer models....

NaN loss in training is critical because it propagates through all parameters and corrupts the model permanently....

Missing gradient clipping causes gradient explosion in RNNs, transformers, and GANs....

Mixed precision loss scale issues occur when GradScaler doesn't update properly, causing underflow or overflow in FP16 training....

Learning rate too high causes training instability, loss divergence, or NaN loss....

Gradient explosion causes loss spikes, NaN loss, and training instability, especially in RNNs and deep networks....

Loss not decreasing indicates fundamental training issues: wrong LR, broken model, bad data, or wrong loss function....

Cosine annealing issues arise from incorrect min_lr, T_max, or warm restarts that cause poor convergence....

Learning rate too low causes slow convergence, plateau at high loss, or training to appear stuck....

Mixed precision overflow occurs in FP16 training when values exceed FP16 range (65504), causing inf/NaN....

Different random seeds across runs cause non-reproducible results, making experiments hard to compare....

Generic runtime errors can indicate various issues from code bugs to hardware problems....

torch.compile errors crash training when the compiled graph has issues with the model or hardware....

PyTorch JIT compilation errors crash training when the script or model uses unsupported features....

AdamW weight decay misconfiguration causes poor generalization or unstable training....

Adam epsilon hyperparameter issues cause training instability or poor convergence....

SGD momentum misconfiguration causes training to oscillate, diverge, or converge slowly....

EMA (Exponential Moving Average) checkpoint issues cause problems with model averaging across checkpoints....

LR warmup-decay schedule issues cause training instability at transitions between phases....

Wrong weight initialization causes training instability with NaN losses, slow convergence, or dead neurons....

EMA (Exponential Moving Average) decay values that are too high or too low cause poor model averaging....

Gradient accumulation misuse causes incorrect gradient updates or memory issues....

Persistent fp16 overflow drives the dynamic loss scale down to its minimum and DeepSpeed aborts. Typically a bf16-pretrained model being fine-tuned in fp16 on hardware without bf16....

DeepSpeed's bf16 training can trigger 'assert all_groups_norm > 0' because bf16 gradient norms can underflow to zero. This is a numerical precision issue specific to bf16's limited range. Denpex detec...

Enabling both overlap_comm and contiguous_gradients in DeepSpeed ZeRO-3 causes gradient buffer reuse races that produce NaN losses. This is a known DeepSpeed bug where the communication overlap reads ...

A CUDA kernel hit a device-side assertion. Almost always an out-of-bounds index into an embedding, loss, or gather/scatter op. Because CUDA is asynchronous, the reported stack trace points at an unrel...

When fine-tuning with Unsloth + TRL using sample packing or padding-free training, the trainer warns that the attention implementation is not flash_attention_2 even though it was configured. Flattened...

Multi-GPU fine-tuning with Unsloth breaks when the Transformers default average_tokens_across_devices=True multiplies the loss by num_processes. The Unsloth fused-loss backward does not expect that sc...

GRPO training that feeds vLLM-generated tensors into the trainable graph fails with 'Inference tensors cannot be saved for backward'. vLLM produces tensors under torch.inference_mode(), which cannot e...

Fine-tuning Gemma 3 with Unsloth crashes with AttributeError: 'Gemma3ModelOutputWithPast' object has no attribute 'loss'. A transformers change to the Gemma3 output class outran Unsloth_zoo's patch, w...

Megatron-LM training hangs at startup while compiling fused kernels (right after 'using torch.float16 for parameters ...') and never proceeds. A stale or contended megatron/fused_kernels/build directo...

Enabling PyTorch Lightning's profiler raises 'ValueError: Another profiling tool is already active'. Python's cProfile allows only one active profiler per thread, so a second cProfile-based profiler (...

Training a Lightning model with torch.compile crashes in the logger connector with a 'called self.log twice with different arguments' MisconfigurationException raised inside the compiled region. Dynam...

DeepSpeed (ZeRO offload) training with HuggingFace Trainer auto_find_batch_size=True hangs right after model load at 0/N steps. The batch-size auto-search re-initializes the engine and desynchronizes ...

Gradient explosions cause loss to diverge to NaN. Denpex traces the norm spike to the layer and step....

NaN loss corrupts training state. Denpex traces NaN propagation to the originating layer....

Weight divergence silently corrupts distributed training. Denpex detects divergence by comparing per-rank weight snapshots....

Infinity loss or weights occur when loss values overflow to infinity, often from numerical instability in mixed-precision training....

Zero gradients stall training when ReLU neurons die or gradient flow is broken in the network....

Loss plateaus occur when training stops making progress, often due to suboptimal hyperparameters or model architecture issues....

Gradient norms explode suddenly while loss remains normal because the cross-rank reduction itself is corrupting values. In-network reduction (SHARP) faults, NCCL data corruption over a marginal link, ...

indexSelectLargeIndex: srcIndex < srcSelectDimSize assertion fires when input token ids exceed the embedding table size. The standard outcome of pairing a tokenizer that has added tokens with a model ...

FP8 training with delayed scaling produces NaNs when an activation outlier saturates the scaling factor (amax history too short / margin too small). Loss and gradients are healthy until the exact step...

The AMP gradient scaler reached a state where every gradient was zero or non-finite, so the scale update divided by zero. The scaler is reporting a numerical collapse upstream, not causing one. This e...

The FP8 attention scale or its reciprocal left the representable E4M3 range, so the kernel produced non-finite output. This is a scale-calibration failure before it is a general training-divergence fa...

LayerNorm produced non-finite gradients because the input variance underflowed. The backward pass divides by sqrt(var + eps); in FP16 a near-constant activation drives variance below representable ran...

Training loss became NaN. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent numerics failures....

Lambda Labs' us-south-3 region experienced a complete network outage when both redundant ISPs simultaneously stopped advertising BGP routes. The primary fiber link was damaged the prior day, shifting ...

A routine UPS-related maintenance procedure in Lambda's us-south-1 region unexpectedly affected power delivery to cooling systems, causing networking equipment and other hardware to either power off o...

Crusoe's AutoClusters system implements a circuit breaker pattern that pauses automatic node replacement when failure patterns look anomalous. If too many nodes in a cluster fail within a short window...

RunPod discovered an issue affecting pods using volume disk or network storage in their CA-MTL-1 region. When executing commands on storage-backed files, processes would hang indefinitely even though ...

The Hugging Face Hub suffered an extended 36-hour outage when repeated requests to space_info_cache and datasets_info_cache MongoDB collections overloaded the database cluster. A script repeatedly que...

Anyscale users experienced GPU starvation when CPU-heavy Ray actors consumed all available CPU slots on GPU nodes, preventing GPU-dependent actors from ever being scheduled. A SpectrogramExtractor wit...

Lambda's us-east-3 region, serving exclusively GH200 instances, experienced intermittent network connectivity caused by a host component kernel panic. Nodes would crash and become unreachable until po...

Ray Data auto-scaling fails when operator resource requirements exceed available cluster resources, throwing ActorUnschedulableError. The specific case documented by Anyscale involved a MapBatches ope...

An AWS us-east-1 outage degraded RunPod's control plane, making the UI, API, and serverless coordination unavailable for GPU users. However, running pod workloads remained fully operational since GPU ...

During HuggingFace's migration from Git LFS to Xet storage backend, migration worker pods filled their ephemeral disks with orphaned temporary shard files. The shard files were first written to /tmp a...

In large-scale distributed training, NCCL errors are almost always the surface symptom rather than the root cause. The actual failure originates at a lower infrastructure layer: GPU hardware fault, st...

RAID storage failures cause data loss and training interruption when storage media degrades....

SLURM cgroup limits restrict resources (CPU, memory, GPU) for jobs, causing OOM kills or throttling when exceeded....

Kubernetes GPU pods can stay in Pending state when GPU resources are not available or configured incorrectly....

nvidia-smi is missing or broken when NVIDIA driver is not properly installed, blocking GPU access entirely....

GPU temperature throttling reduces clock speed and performance when GPUs overheat, sometimes causing training failures....

Cluster shared storage (NFS, Lustre, GPFS) can be a major bottleneck for distributed training data loading....

MIG (Multi-Instance GPU) and MPS (Multi-Process Service) both partition GPU resources, causing conflicts when used together....

GPU TDP and power limit configuration affects performance, thermals, and energy efficiency of training workloads....

SLURM time limit causes training jobs to be killed when they exceed their requested walltime, losing progress....

Container time drift occurs when container time differs from host time, causing TLS failures and timestamp issues....

DNS resolution failures prevent distributed training from finding nodes, downloading from HF Hub, or connecting to services....

Ray task timeouts occur when tasks take longer than the configured timeout....

Ray Dataset OOM occurs when the dataset pipeline uses more memory than available in Ray object store or worker memory....

NFS stalls freeze training when the storage server becomes unresponsive....

Cloud storage throttling slows or fails training when too many requests hit the storage service....

SLURM node OOM kills training when the requested memory exceeds the SLURM memory limit....

Cloud GPU quota limits prevent spinning up new training instances when the account has exhausted its allocated GPU quota....

GPU scheduling delays cause jobs to wait in queue for GPU resources to become available....

A physical link on the parameter (backend) network plane goes down, isolating a node's RDMA path and stalling distributed collectives. One of the most common hardware faults in production traces....

A network interface degrades. Negotiating a lower link speed, disappearing, or reporting GID errors. Silently throttling RDMA collective throughput across the job....

A node power supply fails or loses redundancy, risking a hard node-down event mid-training; sensor telemetry gives early warning before the node drops....

A cooling fan hits a critical speed/fault or loses redundancy, leading to rising GPU temperatures and thermal throttling if not addressed....

Checkpoint writes stall because the NFS RPC layer saturates its slot table, producing a 'bandwidth paradox' where the network is barely utilized yet I/O is the bottleneck....

Docker containers default to 64MB of /dev/shm, which is far too small for NCCL shared memory inter-process communication. Training crashes with 'posix_fallocate failed: No space left on device'. Denpe...

Full disks crash training by preventing checkpoint writes....

Kubernetes OOM kills report the process name (e.g., 'python' or 'java') rather than the pod name, making it highly difficult to trace which specific distributed training job caused a cluster-wide memo...

When mmfsd degrades. Token-manager overload, quorum loss, or internal deadlock. Mounts stall cluster-wide and training processes block in D-state on I/O. Jobs neither progress nor die, mimicking silen...

Azure Blob returns 503 "server busy" and operation timeouts when per-account/partition limits are exceeded by dataset streaming or parallel checkpoint uploads. SDK surfaces vary (ServiceResponseError,...

The NCCL bootstrap routine binds to a virtual ethernet interface (veth) created by Docker/Kubernetes, which lacks routability to peer nodes. TCP connection attempts silently blackhole....

A container gets 64 MB of shared memory by default. PyTorch and NCCL use that space to move data between the worker processes of a tensor-parallel deployment, so a model that runs on one GPU fails to ...

DKMS rebuilds out-of-tree modules against each new kernel, but it can only recompile the source it was given. When the kernel removes or renames an internal function the module calls, the rebuild fail...

The EFA provider reported a disconnected endpoint while polling the completion queue. A peer process exit, security-group mismatch, EFA attachment problem, or libfabric transport fault interrupted the...

RoCE MTU Mismatch - Packets are being dropped because the node MTU exceeds the switch MTU. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cloud failures....

A FlashInfer paged-prefill kernel accessed an invalid GPU address. Common owners are malformed page metadata, a shape or dtype unsupported by the selected kernel, or a binary built for an incompatible...

FP8 matrix multiply was requested on a GPU without hardware FP8 support. torch._scaled_mm requires compute capability 8.9 or newer (Ada, Hopper, Blackwell); Ampere (sm_80/sm_86) has no FP8 tensor core...

Insufficient nvidia.com/gpu resources, scheduler cannot fit pod on any node. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent k8s failures....

The kubelet evicted the pod for exceeding its ephemeral-storage limit. On training pods this is almost always checkpoint or dataset cache written to the container filesystem instead of to a mounted vo...

MIG partition not accessible. MIG mode off or GPU instance misconfigured. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent mig-vgpu failures....

The GPU Operator driver daemonset could not build the NVIDIA kernel module for the running kernel. Until it succeeds the node has no driver, so every GPU pod stays Pending while the node otherwise loo...

The requested MIG profile does not fit alongside the instances already configured. MIG placements are constrained to fixed slot positions, so a set of profiles that sums to the right memory can still ...

The vGPU guest could not reach its Delegated License Service. NVIDIA vGPU throttles clocks severely when unlicensed, so the symptom users report is "the GPU got slow", not "a license expired". This en...

The operating system or container memory limit killed the Raylet, which made the cluster unavailable. This is node memory exhaustion, not an object-store eviction or a recoverable worker-only failure....

Ray could not prepare the job runtime environment before its setup deadline. Package download, dependency installation, or working-directory upload is stalled or too large on at least one node. This e...

SGLang failed to acquire a radix-cache page lock while dynamic prefix eviction was mutating the cache. Extreme token pressure can expose the race, but repeated failure at moderate pressure indicates a...

slurmstepd could not create the memory cgroup for the step. The requested amount exceeds what the node can back, or the cgroup hierarchy is misconfigured, the job never starts rather than being killed...

Slurm drained the node because its NVML health check timed out while the GPU also reported uncorrectable ECC evidence. The timeout is a node-health failure, not a reason to return the GPU to service a...

Slurm step memory cgroup limit hit during NCCL all-gather in FSDP due to memory spike. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent slurm failures....

TensorRT-LLM ran out of paged KV-cache blocks. The block pool is fixed at engine build/startup, so concurrent sequences × their lengths exceeded what was reserved, commonly triggered by beam search or...

Triton's dynamic batcher held requests longer than max_queue_delay_microseconds and dropped them. The server is saturated: arrival rate exceeds what the model instances can clear, so the queue never d...

A TensorRT engine (a .plan/.engine file) was built against a different CUDA/TensorRT version than the one the server is running. TensorRT engines are not portable across versions, they are compiled ar...

The TensorRT-LLM engine build ran out of HOST memory, not GPU memory. Quantization holds full-precision weights in RAM while it converts them, so a 70B model can need well over 100 GB of system memory...

A LoRA adapter was requested whose rank is larger than the engine reserved at startup. vLLM preallocates LoRA slots sized by max_lora_rank, so a higher-rank adapter cannot be loaded into the running e...

vLLM async engine crashed, a worker process exited unexpectedly. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent inference failures....

vLLM fails to initialize due to pre-allocating too much memory for cache blocks when CUDA graphs are enabled or `--max-num-seqs` is set too high. This entry explains how to confirm the cause, apply th...

A vLLM paged-attention kernel accessed an invalid GPU address. The immediate owner can be corrupted block-table metadata, an unsupported shape or dtype, or a binary compatibility defect in the selecte...

NCCL broadcast operations can hang when one rank fails to participate in the collective....

DDP training hangs at the end of an epoch when one rank exhausts its data partition before others....

One DDP rank becomes unresponsive and stalls all ranks waiting for gradient sync....

FSDP flat parameter errors occur when the flattened parameter management fails during sharding or unsharding operations....

FSDP mixed precision errors arise when parameter precision settings conflict between FSDP wrapping and autocast....

Tensor parallel errors occur when model layer splitting across GPUs fails due to dimension mismatches or communication issues....

DeepSpeed initialization fails when configuration is invalid or incompatible with the model....

PyTorch distributed errors prevent DDP and FSDP from functioning correctly....

NCCL watchdog settings control how long collectives can stall before timing out. Wrong settings cause premature or delayed failure detection....

When one rank stalls, NCCL's synchronous collectives freeze every rank, masking which GPU actually failed. Localizing the culprit needs divergence signals, not the NCCL stack trace....

The configured world size does not match the number of ranks that actually connect, so NCCL initialization hangs or errors during the communication-setup stage....

DeepSpeed ZeRO-3 with Mixture-of-Experts (MoE) models hangs during training because MoE blocks need to be marked as leaf modules for ZeRO-3 parameter gathering. Without this marking, ZeRO-3 attempts t...

DeepSpeed ZeRO-3 crashes with 'dict object has no attribute _in_forward' when used with PyTorch 2.5+, which changed the internal _parameters attribute from a dict to a different type. Denpex detects t...

DeepSpeed ZeRO-3 crashes with UnboundLocalError in partition_parameters.py when the model has very small parameters (fewer elements than the number of GPUs). The partition math divides a small paramet...

After unwrapping an FSDP2 (fully_shard) model the parameters are still DTensors, so save_pretrained or a plain state_dict produces invalid storage or sharded tensors. You must gather a full, unsharded...

Combining a bitsandbytes 8-bit optimizer with DeepSpeed ZeRO offload crashes with 'Error invalid configuration argument at line 216 in file .../bitsandbytes/csrc/ops.cu'. The two optimizer-state manag...

FSDP combined with QLoRA fails to build its flat parameter: 'Must flatten tensors with uniform dtype but got torch.float32 and torch.bfloat16'. QLoRA keeps some parameters in float32 while the base is...

After upgrading to DeepSpeed >0.14.0 (e.g. 0.14.2), ZeRO-3 / optimizer-offload training crashes with 'Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!'. A...

Model weights are not written by accelerator.save_state when the model was set up with accelerator.prepare_model(model) instead of accelerator.prepare(...). prepare_model only wraps the model for dist...

DeepSpeed ZeRO-3 errors during parameter allgather when the model holds parameters in more than one dtype (e.g. bf16 base with some fp32 modules). Older ZeRO-3 assumed a single dtype for coalesced all...

DeepSpeed pipeline-parallel training (PipelineModule) hangs mid-run when microbatch input shapes vary. The PP engine caches the first microbatch's tensor shapes for point-to-point send/recv buffers, s...

DDP setup errors prevent distributed training from initializing correctly....

DDP port conflicts occur when the chosen master port is already in use by another process....

DDP NaN detection on a single rank often indicates inconsistent data or hardware between ranks....

DDP OOM during allreduce occurs when NCCL communication buffers exceed available GPU memory....

FSDP state_dict missing keys errors prevent checkpoint saving or loading in sharded models. Denpex identifies the source of key mismatches....

FSDP all-gather timeouts stall training when sharded parameters cannot be collected from distributed ranks....

DeepSpeed OOM errors crash ZeRO-optimized training when memory optimization settings don't match model architecture....

DeepSpeed ZeRO hangs stall training when gradient synchronization stalls or parameter offload produces deadlocks....

DeepSpeed initialization fails when configuration is invalid or incompatible with the model....

Megatron-LM initialization errors prevent large model training from starting due to configuration mismatches....

Torchrun's approach to spawning jobs conflicts with SLURM's process management, resulting in hung NCCL initialization, rank mismapping, and job allocation failures....

A single GPU drops in performance without throwing an error, causing all other GPUs in the collective to wait. This silently degrades cluster-wide throughput....

NCCL timeout occurs while gathering the state dictionary or broadcasting during a checkpoint save or restore operation....

ChildFailedError is torchelastic's envelope around a dead worker. The exitcode/signal in its table is the actual signal: -9 host OOM-kill, -11 segfault, -6 abort (often CUDA), 1 python exception whose...

Data-dependent control flow causes the JIT compiler to generate divergent execution graphs across ranks based on local data variance. Ranks attempt to execute different collective operations....

Hugging Face Accelerate integration drops the `InitProcessGroupKwargs` timeout parameter when DeepSpeed owns the distributed initialization process. The PyTorch watchdog defaults back to 600 seconds....

DDP's gradient reducer arms itself for exactly one backward pass per forward and disarms once that pass is finalised. A second backward, or a backward whose forward was run under a different synchroni...

A language model whose output projection shares its tensor with the input embedding fails to shard. When fully_shard rebuilds parameters as DTensors, the shared tensor appears once rather than twice, ...

Choosing not to auto-wrap is legitimate and leaves the wrapping policy unset. Activation checkpointing then tries to apply that policy to decide which submodules to checkpoint, calls the empty value, ...

Sharding and CPU offloading both move parameters, and each assumes it is the component deciding where a parameter lives. Enabling them together produces a failure during setup or the first step, in a ...

Connecting to a cluster starts failing intermittently on ARM64 nodes after a transport library is upgraded, while the identical image on x86 continues to work. The client reports that the server abort...

A quantised mixture-of-experts checkpoint loads on a single device and fails under sharded loading. Materialising a module from the placeholder device reads each parameter by name, and the quantised e...

The actor blocked waiting for a ZeRO-3 parameter gather owned by the critic. Two ZeRO-3 models in one process share a parameter coordinator; if their forward passes interleave, each can wait on a coll...

FP16 gradient overflow forced the loss scaler to halve repeatedly. Occasional skipped steps early in training are normal; a scale that keeps collapsing means gradients are genuinely exploding or the m...

DeepSpeed ZeRO-Offload filled the NVMe filesystem used for optimizer or parameter swap. The failed aio_write means the offload state is incomplete, even if the training process reports a later optimiz...

ZeRO-2 batches gradients into a fixed communication bucket before reduce-scatter. A gradient tensor larger than the bucket, or too many arriving at once, exhausts it and the reduction cannot proceed. ...

ZeRO-3 keeps parameters sharded until they are gathered for use. Code touched a parameter outside a gather context, so it was still partitioned, typically custom forward logic, weight tying, or an ini...

The reference model and the policy model have different vocabulary sizes, so their logits cannot be compared. DPO scores the policy against the reference token by token, a vocab difference of even a f...

An FSDP gradient reduce-scatter did not complete within the collective timeout. Like any collective timeout this names the ranks that were WAITING, not the one that failed to arrive, one rank left the...

FSDP ran out of memory gathering a layer's full weights for the forward pass. Sharding reduces STEADY-STATE memory, but each layer is briefly unsharded to compute, so the peak is set by the largest si...

jax.distributed.initialize() timed out waiting for all processes to check in with the coordinator. One or more workers never reached the rendezvous, they crashed at startup, cannot reach the coordinat...

XLA could not allocate device memory. JAX preallocates ~75% of VRAM at first use by default, so an OOM here often reflects the preallocation policy or memory fragmentation across recompilations rather...

A pipeline stage waited for activations from its neighbour and timed out. In pipeline parallelism a single slow or dead stage stalls the whole pipeline, so the rank that reports the timeout is the VIC...

The rotary embedding dimension does not divide evenly into the attention head dimension. This is a configuration arithmetic error, caught at construction: hidden_size / num_attention_heads must be com...

Sequence parallelism was enabled while tensor parallel size remained 1. Megatron partitions sequence work across the tensor-parallel group, so there is no group to partition across in this configurati...

NCCL collective timed out. One rank stopped participating. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent pytorch-fsdp-ddp failures....

Ray could not reserve the requested placement group before the timeout. The cluster does not have a set of nodes that simultaneously satisfies the bundle layout, commonly an RLHF setup asking for colo...

GPU ran out of memory during training. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent pytorch-fsdp-ddp failures....

Tensors on different devices (CPU vs GPU, or different GPUs) in an operation that requires same-device tensors. This entry explains how to confirm the cause, apply the fix, and separate it from adjace...

Something still attached to the autograd graph was handed to the checkpoint writer. Saving it would serialise the graph as well as the values, so torch refuses rather than writing a checkpoint that ca...

torchrun ChildFailedError, a worker process died and torchelastic is reporting the WRAPPER error. The real failure is in the per-rank traceback above/below this block (exitcode -9 = OOM kill, 1 = pyth...

TRL PPO could not keep the policy, reference model, value head, activations, and optimizer state within the available VRAM. The simultaneous model residency makes this different from a single-model ba...

Augmentation pipeline errors cause inconsistent or corrupted training data when transforms fail or produce unexpected outputs....

TorchData (torch.utils.data.datapipes) errors occur when complex data pipelines have configuration or compatibility issues....

HDF5 file corruption causes training to fail with I/O errors or silently load incorrect data....

Apache Arrow IPC errors occur when serialized Arrow data is corrupted or has version incompatibilities....

TFRecord and tf.data errors occur when TensorFlow data pipeline has issues with the training data or pipeline configuration....

ImageNet preprocessing mismatches cause pretrained models to underperform because input normalization differs from training....

WebDataset TAR file corruption causes data loading failures and missing samples during training....

Overly aggressive data augmentations hurt model performance by distorting critical features in training data....

Memory-mapped (mmap) file handles are exhausted when datasets have many large files, causing data loading failures....

Small shuffle buffers cause poor data ordering during training, hurting model generalization....

Tokenizer padding mismatches cause data loader errors or poor model performance when padding side or token ID differs....

Audio sample rate mismatches between training data and pretrained models cause poor ASR/TTS performance....

Parquet schema mismatches occur when reading parquet files with different schemas than expected by the loading code....

Streaming data errors occur when loading data from cloud storage or remote endpoints fails mid-training....

Video codec mismatches between training data and pretrained video models cause loading failures or poor performance....

TFRecord corrupted shard errors occur when individual TFRecord files are truncated or have inconsistent examples....

LMDB corruption occurs when LMDB files are not properly closed, transactions interrupted, or map_size exceeded....

RAG embedding mismatches cause retrieval to return irrelevant documents when embedding model differs between indexing and query....

Multi-label class imbalance causes poor performance on rare labels and biased predictions toward frequent labels....

Contrastive learning augmentation pipelines must be carefully tuned to create useful positive pairs without destroying semantic content....

Image resize artifacts degrade model accuracy when resize method (BILINEAR, BICUBIC, LANCZOS) doesn't match pretrained model expectations....

LLM tokenization truncation silently cuts long sequences, causing loss of information in long documents....