Uneven Pipeline Stage Partitioning as Primary Straggler Cause in LLM Training

In a five-month trace analysis of ByteDance's LLM training cluster, uneven pipeline stage partitioning was identified as the most prevalent cause of training stragglers, affecting 39.3% of jobs. The last pipeline stage containing loss computation layers is significantly heavier than earlier stages, creating systematic imbalance that propagates cascading bubbles through the entire pipeline.

Uneven Pipeline Stage Partitioning as Primary Straggler Cause in LLM Training is a Fail-Slow failure seen during ML training runs. In a five-month trace analysis of ByteDance's LLM training cluster, uneven pipeline stage partitioning was identified as the most prevalent cause of training stragglers, affecting 39.3% of jobs. The last pipeline stage containing loss computation layers is significantly heavier than earlier stages, creating systematic imbalance that propagates cascading bubbles through the entire pipeline. Common tags: Bytedance, Straggler, Pipeline Parallel, Megatron Lm.

Loss computation layers (lm_head, cross-entropy) have significantly more parameters and compute than transformer layers. Static pipeline partitioning assigns equal layer counts per stage without considering per-layer compute asymmetry. Last stage's additional work creates a standing straggler that cannot be recovered by faster earlier stages. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training throughput drops 10-120% compared to ideal straggler-free execution
• Specific pipeline stages consistently finish iterations 2-3x slower than others
• Profiling shows loss-related layers dominating per-stage compute time

• Megatron-LM / DeepSpeed pipeline-parallel training on 128+ GPUs
• Hybrid parallel training combining data, tensor, and pipeline parallelism
• LLMs with heavy loss-head layers (cross-entropy, lm_head)

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Per-microbatch iteration time shows high variance correlated with pipeline stage index
• ✓Verified signal present: NCCL all-reduce finish times cluster around the slowest stage rank
• ✓Verified signal present: Pipeline bubble ratio exceeds theoretical minimum calculated from stage count
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

DeepSpeed changes when parameters, gradients and optimizer state are created, partitioned, gathered and offloaded. The hub separates ZeRO, memory, checkpoint and pipeline failures by lifecycle phase.

• Loss computation layers (lm_head, cross-entropy) have significantly more parameters and compute than transformer layers
• Static pipeline partitioning assigns equal layer counts per stage without considering per-layer compute asymmetry
• Last stage's additional work creates a standing straggler that cannot be recovered by faster earlier stages