PyTorch Caching Allocator Memory Fragmentation Causing False Straggler Slowdown

ByteDance's production straggler analysis identified PyTorch's CUDA caching allocator memory fragmentation as a previously unreported cause of training stragglers. Over long-running jobs, the allocator fragments GPU memory into small non-contiguous blocks, causing kernel launches that require large contiguous allocations to trigger expensive defragmentation cycles on specific ranks.

PyTorch Caching Allocator Memory Fragmentation Causing False Straggler Slowdown is a Reliability failure seen during ML training runs. ByteDance's production straggler analysis identified PyTorch's CUDA caching allocator memory fragmentation as a previously unreported cause of training stragglers. Over long-running jobs, the allocator fragments GPU memory into small non-contiguous blocks, causing kernel launches that require large contiguous allocations to trigger expensive defragmentation cycles on specific ranks. Common tags: Bytedance, Memory Fragmentation, Caching Allocator, Straggler.

PyTorch CUDA caching allocator segments memory into blocks but cannot coalesce blocks of different sizes. Variable sequence lengths and dynamic tensor shapes create allocation patterns that leave small gaps between active blocks. Fragmentation accumulates over time as training progresses through different data patterns. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training throughput degrades gradually over time on specific ranks but not others
• torch.cuda.memory_summary() shows >30% fragmentation on affected GPUs
• GPU utilization on affected ranks drops below 60% while other ranks maintain >90%

• PyTorch training on H100/A100 with variable-length sequences or dynamic shapes
• Jobs running >48 hours on a single GPU allocation
• Models with many tensor parallelism or pipeline parallelism shards

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: CudaMalloc retry count increases on affected rank: verbose logging via `CUDA_LAUNCH_BLOCKING=1` shows allocation retries
• ✓Verified signal present: Iteration time on a single rank increases by 15-30% over the course of a week
• ✓Verified signal present: torch.cuda.memory_snapshot() shows thousands of small segments with few large contiguous blocks
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

CUDA reports errors asynchronously, so the traceback usually points at whatever line synchronised next rather than the one at fault. The hub covers every common CUDA error and how to make it report honestly.

• PyTorch CUDA caching allocator segments memory into blocks but cannot coalesce blocks of different sizes
• Variable sequence lengths and dynamic tensor shapes create allocation patterns that leave small gaps between active blocks
• Fragmentation accumulates over time as training progresses through different data patterns