Memory error references

Browse 77 published references for memory failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• Activation Checkpoint Misuse

  Activation checkpointing misuse causes either OOM (not enough checkpointing) or slow training (too much checkpointing).

• Activation Checkpointing Compatibility

  Activation checkpointing compatibility issues arise when using torch.compile, FSDP, or DDP with checkpointing.

• Activation Distillation Memory

  Activation distillation memory costs grow with teacher model size and student hidden dimension matching requirements.

• Activation Memory Explosion from Quadratic Attention

  Standard multi-head dot-product attention computes an attention matrix of size (batch_size, num_heads, seq_length, seq_length). The memory complexity is O(N^2) with respect to the sequence length. As sequence length increases, the intermediate activations saved for the backward pass become excessively large and exhaust VRAM.

• Activation Memory Spike

  Activation memory spikes during specific operations cause transient OOM even when steady-state memory fits.

• Checkpoint Loading Memory

  Loading checkpoints can temporarily double memory usage because both old and new model states exist in memory.

• Computation Graph Memory Leak via Loss Accumulation

  The developer accumulates the loss tensor across batches for logging (e.g., `total_loss += loss`). Because `loss` is a PyTorch tensor attached to the computation graph, accumulating it keeps the entire computation graph for every batch in memory, preventing garbage collection.

• CPU Offloading Overhead

  CPU offloading trades GPU memory for PCIe bandwidth. Parameters, gradients or optimizer state cross the bus every step, and when the transfer cannot hide behind compute the step becomes bound by the link rather than the GPU.

• CPU RAM Exhaustion During Training

  CPU RAM exhaustion during training kills the process or causes swapping that drastically slows training.

• CUDA Caching Allocator Fragmentation

  CUDA caching allocator fragmentation causes OOM despite enough total free memory, due to non-contiguous blocks.

• CUDA Caching Allocator Issue

  CUDA caching allocator issues cause memory not being released back to the GPU when expected.

• CUDA Context Leak

  CUDA context leaks occur when CUDA contexts are not properly destroyed, accumulating GPU memory across processes.

• CUDA Graph Memory Trap

  CUDA Graphs capture memory allocations that are hard to release, causing memory leaks across graph replays.

• CUDA Memory Allocation Failed

  CUDA memory allocation fails when the requested memory block cannot be allocated, often due to fragmentation or insufficient total memory.

• CUDA Memory Fragmentation OOM

  GPU memory fragmentation causes CUDA OOM even when total free memory exceeds the allocation request. PyTorch caching allocator cannot find a single contiguous block large enough, leading to a false OOM.

• CUDA Memory Leak

  CUDA memory leaks accumulate over training, eventually causing OOM. Denpex tracks allocation patterns to identify leaks.

• CUDA OOM from Caching Allocator Fragmentation

  FSDP performs frequent allocations and deallocations of varying sizes (flattening, padding, un-sharding, sharding). This usage pattern severely fragments the CUDA caching allocator. Memory gets trapped in blocks that cannot be merged, meaning PyTorch cannot find a contiguous block large enough for a new tensor, even though total free VRAM is sufficient.

• CUDA Out of Memory

  CUDA out of memory is the most common training error, occurring when GPU memory is exhausted.

• CUDA out of memory during sentence_transformers encode sequence length batch size

  Sentence Transformers exceeded VRAM during encoding because the token count, batch size, output handling, and other GPU residents exceeded the model memory budget. Sequence length can dominate even when the document count looks small. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent vectordb failures.

• CUDA Shared Memory Limit Exceeded

  CUDA shared memory limits are reached when kernels use too much shared memory per block.

• CUDA Unified Memory Error

  CUDA Unified Memory (UVM) errors cause page faults and performance issues when memory oversubscription occurs.

• cudaHostAlloc out of memory pinned memory check ulimit -l max locked memory

  A large pinned host-memory allocation exceeds the host pinning path; the generic CUDA debugging footer is not an observed device assertion. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cuda-ipc-shm failures.

• cudaIpcGetMemHandle returned error out of memory

  CUDA could not export an interprocess memory handle for the allocation. Device memory pressure, an ineligible allocation, or exhausted IPC and driver bookkeeping can cause this before the receiving process maps anything. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cuda-ipc-shm failures.

• cuDNN Benchmark Memory: Check PyTorch Convolution OOM

  Investigate PyTorch convolution OOM with cuDNN benchmark enabled. Compare input shapes, peak memory and algorithm selection before changing the workload.

• Dataset Cache Memory

  Dataset cache memory usage grows when transformations or augmentations are applied before batching.

• DeepSpeed FusedAdam Illegal Memory Access on H100

  DeepSpeed FusedAdam optimizer triggers CUDA illegal memory access errors on NVIDIA H100 GPUs. This is a compatibility issue between FusedAdam's CUDA kernels and H100's SM90 architecture. Denpex detects the H100+FusedAdam combination and recommends the fallback optimizer.

• DeepSpeed OOM

  DeepSpeed OOM errors occur when ZeRO partitioning, CPU offload, or activation partitioning is misconfigured.

• DeepSpeed ZeRO GPU Memory Not Freed After Training (hooks leak)

  GPU memory is not released after trainer.train()/engine teardown, so repeated training calls (CV folds, hyperparameter sweeps) leak until OOM. DeepSpeed left gradient-accumulation/backward hooks attached after engine destroy, pinning the model and optimizer state. Upgrade and explicitly free memory between runs.

• DeepSpeed ZeRO Offload Host-RAM OOM (SIGKILL -9)

  ZeRO CPU/NVMe offload materializes parameters and optimizer states in host RAM on every rank, exhausting system memory and getting the process OOM-killed with no Python traceback.

• DeepSpeed ZeRO-3 High GPU Memory / OOM Loading a Large Model Without zero.Init

  Fine-tuning a large model (e.g. Flan-T5-XXL 11B) with ZeRO-3 uses far more GPU memory than expected or OOMs at load because the model was constructed outside DeepSpeed's zero.Init context. Every rank materializes the full dense model before partitioning. Build the model under zero.Init (or keep HfDeepSpeedConfig alive before from_pretrained).

• DeepSpeed ZeRO-3 Parameter Offload Illegal Access

  ZeRO-3 partitions model parameters and offloads them to CPU memory. When a custom operation (like a custom regularizer or un-traced optimizer step) tries to access a parameter tensor directly, it passes a CPU pointer to a CUDA kernel, or attempts to read an un-gathered partitioned tensor, resulting in a segfault or illegal memory access.

• Embedding Layer Memory

  Embedding layer memory grows with vocabulary size and can dominate total memory for large vocab models.

• Evaluation Loop OOM due to Missing torch.no_grad

  The evaluation loop is executing without `with torch.no_grad():` or `@torch.inference_mode()`. Therefore, PyTorch continues to build the autograd computation graph during evaluation, storing activations. Because `.backward()` is never called, the graph is never freed, leading to a massive memory spike.

• faiss StandardGpuResourcesImpl allocMemory CUDA error out of memory

  FAISS could not allocate GPU memory for the index. A flat GPU index holds every vector in VRAM, so the requirement is vectors × dimensions × 4 bytes plus working space, it scales linearly and unforgivingly with corpus size. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent vectordb failures.

• Flash Attention Memory

  Flash Attention saves memory by not materializing the full attention matrix, but has shape and dtype constraints.

• FlashAttention Max Sequence Length Illegal Access

  When using variable length sequences in FlashAttention (e.g., passing `cu_seqlens`), if the actual sequence length in the data exceeds the `max_seqlen_k` or `max_seqlen_q` parameter passed to the kernel, the highly optimized block-wise memory loader reads outside the allocated KV-cache or attention mask bounds.

• Forward Pass OOM due to Excessive Parameter Prefetching

  DeepSpeed ZeRO-3 uses a `prefetch_bucket_size` (defaulting to a very large number like 50M) to preemptively AllGather upcoming layers into VRAM to hide communication latency. For large models or restricted memory hardware, pulling too many future layers into GPU memory at once overrides the savings of sharding, causing an OOM.

• FSDP All Gather Timeout

  FSDP all-gather operations can timeout when parameters are large or network is slow, causing training to fail.

• GPU Uncorrectable ECC Memory Error (Xid 48/94/95)

  GPU HBM memory experienced an uncorrectable ECC error (double-bit error) that the hardware cannot fix. This indicates physical HBM memory degradation requiring GPU drain and potential RMA.

• Gradient Accumulation Memory Spike

  Gradient accumulation can cause unexpected memory spikes when scaled batch size is large or loss is not properly reduced.

• Gradient Checkpointing Tradeoff

  Gradient checkpointing trades compute for memory; misconfiguration can either not save memory or drastically slow training.

• Host OOM Killed

  Host OOM (out of memory) kills training processes when system RAM is exhausted, often silently.

• Host RAM Exhaustion via ZeRO Offload Pinned Memory

  When using ZeRO-Offload (CPU offloading for parameters/optimizer states), DeepSpeed enables `pin_memory: true` by default. Pinned memory (page-locked memory) cannot be swapped to disk. Offloading large states into pinned memory rapidly exhausts available physical RAM, triggering the Linux OOM killer.

• Host RAM OOM-Kill of DataLoader Workers Misread as GPU OOM

  The kernel oom-killer (or cgroup limit) kills python workers whose RSS grew unbounded. Shard caches, tokenized-in-RAM datasets, prefetch queues. Surface strings ("Out of memory", "Killed") route engineers to GPU OOM playbooks that cannot help.

• Inference Memory Leak

  Inference memory leaks cause growing GPU memory usage during serving, eventually OOM-ing the server.

• KV Cache Memory Growth

  KV cache memory grows with sequence length in transformer inference and some training setups, causing OOM at long contexts.

• Memory Fragmentation

  Memory fragmentation causes OOM even when total free memory is sufficient, because no contiguous block is available for the allocation.

• Memory Leak in DataLoader

  Memory leaks in DataLoader workers cause steadily growing memory usage over training epochs.

• Memory Summary Tool Usage

  torch.cuda.memory_summary() helps debug memory issues but output can be overwhelming without understanding it.

• milvus Failed to load vector index segment into GPU memory insufficient VRAM

  Milvus could not load a vector index segment into GPU memory. Segments are loaded whole, so a single oversized segment fails even when total free VRAM across the collection looks sufficient. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent vectordb failures.

• nn.Embedding Out of Bounds Illegal Memory Access

  An index passed to a PyTorch `nn.Embedding` layer exceeds the configured `num_embeddings` (e.g., passing token ID 1000 to an embedding matrix of size 1000, where max valid is 999). Because CUDA executes asynchronously, the out-of-bounds kernel memory violation doesn't throw a Python exception immediately. Instead, the GPU's memory protection fault corrupts the CUDA context, causing a subsequent operation to raise the generic 'illegal memory access' error.

• Optimizer State Memory

  Optimizer state memory (Adam: 2x model size, AdamW: 2x) can exceed model size memory, dominating total usage.

• Optimizer State Memory Explosion

  Optimizers like Adam and AdamW maintain moment states (running averages of gradients and squared gradients) for every parameter. For a model with N parameters, the optimizer state requires 2 * N memory (often in FP32). The initialization of these state tensors happens lazily on the first `optimizer.step()`, suddenly doubling or tripling the memory footprint of the model weights.

• Pinned Memory Overuse

  Pinned (page-locked) memory overuse causes system RAM exhaustion and slow CPU-GPU transfers when over-allocated.

• Pipeline Parallel Memory

  Pipeline parallel memory issues arise from stage imbalance, bubble overhead, and activation storage across micro-batches.

• Python Out of Memory

  Python OOM kills the training process when Python's heap memory is exhausted, even if GPU memory is fine.

• PyTorch Caching Allocator Debug

  Debugging PyTorch's caching allocator helps identify memory issues but requires understanding its behavior.

• PyTorch CUDA Memory Fragmentation

  The PyTorch caching allocator reserves large blocks of memory and splits them for individual tensor allocations. Over time, memory becomes fragmented into many small blocks, preventing the allocation of a new contiguous large block, even if the total free memory across all small blocks is sufficient.

• Rank 0 OOM During ZeRO-3 Checkpoint Save

  By default, DeepSpeed's ZeRO-3 config parameter `gather_16bit_weights_on_model_save` is often set to true. This instructs DeepSpeed to unshard and consolidate all model weights onto a single GPU (Rank 0) to save a standard fp16 model file. For massive models, gathering the entire model state on a single GPU immediately exceeds its VRAM capacity.

• Safetensors Load Error

  Safetensors load errors occur from corrupted files, version mismatches, or architecture mismatches with the loading model.

• Silent Pod GPU OOM Hang

  A CUDA Out Of Memory (OOM) error is an application-level exception thrown by the CUDA runtime to the framework (e.g., PyTorch). Unlike a system RAM OOM where the Linux kernel's OOM-killer sends a SIGKILL (resulting in an 'OOMKilled' pod state), a CUDA OOM does not automatically terminate the process. If the application catches the exception but fails to exit gracefully, the container remains alive in a deadlocked or idle state.

• System Memory OOM on FSDP Checkpoint Save

  When using `FullStateDictConfig(offload_to_cpu=True)` without setting `rank0_only=True`, FSDP instructs every single rank to independently gather and reconstruct the full model parameters into its own CPU memory. For a large model and multiple ranks on a node (e.g., 8 GPUs), this creates 8 redundant copies of the full model in CPU RAM, instantly exhausting system memory.

• Tensor on device meta is not on the expected device: a device mismatch from a parameter never materialised

  A model built with automatic device placement runs until one submodule executes with a parameter still on the placeholder device. Meta tensors carry shape and dtype but no data, so the operation refuses rather than computing on nothing, and the error names a device that is not a device at all.

• Tensor Parallel Memory

  Tensor parallel training splits model across GPUs, but communication buffers and synchronization can still cause OOM.

• Tensor Views Memory Leak

  Tensor views (slices, reshapes, transposes) can hold references to large base tensors, preventing memory release.

• Time-Slicing Cross-Pod VRAM Exhaustion

  Kubernetes GPU time-slicing multiplexes compute execution time, but it provides absolutely zero hardware-level memory isolation. If a physical GPU with 24GB VRAM is time-sliced into 4 logical GPUs, Kubernetes allows scheduling 4 pods to it. If Pod A (the 'noisy neighbor') allocates 20GB of VRAM, Pods B, C, and D will instantly encounter CUDA OOM errors because the physical memory pool is exhausted, despite Kubernetes acting as if they each possess dedicated resources.

• torch.compile Memory

  torch.compile can use additional memory for graph compilation, guard evaluation, and dynamic shape handling.

• torch.multinomial Invalid Probability Memory Access

  When `torch.multinomial` receives a probability distribution tensor containing `NaN` values, negative values, or all zeros, the CUDA kernel fails to build a valid Cumulative Distribution Function (CDF). This causes the sampling kernel to read/write out of bounds when attempting to perform binary search on the corrupted CDF array.

• Transformer Attention Memory

  Transformer attention memory grows quadratically with sequence length; long-context training requires Flash Attention or similar.

• Transformer Cache Memory

  Transformer cache (KV cache, past_key_values) memory grows with sequence length and can cause OOM at inference.

• unable to mmap bytes from /dev/shm No space left on device torch multiprocessing

  PyTorch multiprocessing could not map another shared-memory region because /dev/shm is full or capped too low. This consumes shared memory, not filesystem disk and not GPU VRAM. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cuda-ipc-shm failures.

• UVM fault excessive page migrations between host RAM and GPU HBM cudaMallocManaged

  Managed-memory pages are bouncing between host RAM and GPU HBM faster than either processor can reuse them. The migration traffic saturates the interconnect and can surface as an Xid 31 page fault or a severe stall. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cuda-ipc-shm failures.

• vLLM refuses to start with No available memory for the cache blocks

  vLLM aborts during engine initialisation because, after loading model weights and profiling the activation peak, nothing is left inside its memory budget to allocate a single paged KV cache block. The GPU is frequently not full: the free memory sits outside the fraction vLLM was told it may use.

• vLLM rejects a request because the prompt plus completion exceeds the served context length

  vLLM returns an error for a request whose prompt tokens plus requested completion tokens exceed the context window the server was started with. The served window is not always the one advertised on the model card: it is taken from the model config and can be capped further when KV cache memory is short.

• vLLM runs out of GPU memory during serving after the prefill token budget is raised

  Raising the number of tokens vLLM may batch per step improves time to first token and throughput, and it also raises the peak activation memory a prefill step needs. When that peak collides with the memory already reserved for the KV cache, the failure arrives during serving rather than at startup, after the deployment has looked healthy.

• Watchdog Timeout Caused by Silent Single-Rank OOM

  A single rank hits a CUDA Out of Memory error during the forward or backward pass and crashes or raises an exception. Because the process group isn't cleanly torn down, the surviving ranks continue to the next all_reduce or all_gather and wait indefinitely for the dead rank. The watchdog timeout is the only visible failure on the surviving ranks.

• Xid 31: Memory Page Fault from Process Resource Exhaustion

  Xid 31 occurs when the GPU attempts to access a virtual address that is not mapped in the GPU page tables. While this can be a kernel bug, it frequently happens under severe memory pressure or thrashing where memory is allocated but not properly bound, or an application uses an invalid pointer that falls outside its mapped VRAM range.