CUDA Memory Fragmentation OOM

GPU memory fragmentation causes CUDA OOM even when total free memory exceeds the allocation request. PyTorch caching allocator cannot find a single contiguous block large enough, leading to a false OOM.

CUDA Memory Fragmentation OOM is a GPU Memory Management failure seen during ML training runs. GPU memory fragmentation causes CUDA OOM even when total free memory exceeds the allocation request. PyTorch caching allocator cannot find a single contiguous block large enough, leading to a false OOM. Common tags: Cuda, Oom, Fragmentation, Memory.

PyTorch caching allocator reserves memory in fixed-size segments; freed segments of different sizes create a fragmented heap. A large contiguous allocation (e.g., attention score matrix) cannot fit in any single free segment. Without expandable segments, the allocator cannot grow an existing segment to satisfy the request. Variable sequence length training creates differently-sized activation tensors each iteration, accelerating fragmentation. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• PyTorch reports CUDA out of memory despite total free memory being sufficient
• Error shows Reserved >> Allocated with Free memory still available
• Training fails on large tensor allocations (e.g., attention scores, activation checkpoints)
• Reducing batch size does not always help because the issue is fragmentation, not capacity

• PyTorch training with variable tensor sizes across iterations
• Transformer models with variable sequence lengths
• Long-running training jobs with repeated allocation/deallocation cycles
• Custom CUDA kernels with manual memory management

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: CUDA OOM error showing Reserved: X, Allocated: Y, Free: Z where Free > requested size
• ✓Verified signal present: Error occurs on the same iteration every run (deterministic fragmentation pattern)
• ✓Verified signal present: expandable_segments or memory fragmentation mentioned in error context
• ✓Verified signal present: OOM happens during forward pass on large tensors, not during initialization
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

CUDA reports errors asynchronously, so the traceback usually points at whatever line synchronised next rather than the one at fault. The hub covers every common CUDA error and how to make it report honestly.

• PyTorch caching allocator reserves memory in fixed-size segments; freed segments of different sizes create a fragmented heap
• A large contiguous allocation (e.g., attention score matrix) cannot fit in any single free segment
• Without expandable segments, the allocator cannot grow an existing segment to satisfy the request
• Variable sequence length training creates differently-sized activation tensors each iteration, accelerating fragmentation