Python Garbage Collection Triggering Periodic Training Stragglers in Distributed LLM Training

ByteDance's production trace analysis found that Python's automatic garbage collection causes periodic, transient straggler behavior in distributed LLM training. Full GC passes on long-lived objects (preallocated buffers, tokenizer state, lookup tables) can stall rank computation for 200-500ms per event, enough to delay NCCL collectives and create cascading pipeline bubbles.

Python Garbage Collection Triggering Periodic Training Stragglers in Distributed LLM Training is a Reliability failure seen during ML training runs. ByteDance's production trace analysis found that Python's automatic garbage collection causes periodic, transient straggler behavior in distributed LLM training. Full GC passes on long-lived objects (preallocated buffers, tokenizer state, lookup tables) can stall rank computation for 200-500ms per event, enough to delay NCCL collectives and create cascading pipeline bubbles. Common tags: Bytedance, Python Gc, Straggler, Gc Freeze.

Python's generational GC (ref. counting + cycle detector) triggers full generation-2 scans on all heap objects. Preallocated training buffers, tokenizer state, and lookup tables are long-lived objects that land in oldest generation. Full GC passes walk hundreds of thousands of references in the oldest generation, stalling Python execution. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Every few minutes, a single rank stalls for 200-500ms with no GPU activity
• CPU functions that normally complete in <5ms suddenly take >100ms
• Stalls correlate with Python GC generation 2 (full) collection events

• PyTorch training loops with Python-level data preprocessing
• Large preallocated object pools (tokenizers, embeddings, lookup tables)
• Multi-node training with sensitive NCCL timeout thresholds

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Per-rank iteration time shows regular periodic spikes at 2-5 minute intervals
• ✓Verified signal present: NCCL_DEBUG output shows rank falling behind collective barrier by >200ms
• ✓Verified signal present: gc.get_stats() shows generation 2 collection count increasing during training with collection duration >150ms
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Python's generational GC (ref. counting + cycle detector) triggers full generation-2 scans on all heap objects
• Preallocated training buffers, tokenizer state, and lookup tables are long-lived objects that land in oldest generation
• Full GC passes walk hundreds of thousands of references in the oldest generation, stalling Python execution