Training Stuck No Progress

Training appears stuck with no progress, often caused by deadlocks, infinite loops, or network hangs in distributed training.

Training Stuck No Progress is a Reliability failure seen during ML training runs. Training appears stuck with no progress, often caused by deadlocks, infinite loops, or network hangs in distributed training. Common tags: Stuck, No Progress, Deadlock, Harness.

Deadlock in distributed training. Data loading is the bottleneck. Learning rate too small (no visible progress). No gradient flow (detached graph). All-reduce hang in DDP. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training loss doesn't decrease
• No progress for many iterations
• GPU utilization drops to zero

• Distributed training with hangs
• Complex models with deadlocks
• Long training runs

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Loss is flat for many steps
• ✓Verified signal present: Validation metrics don't improve
• ✓Verified signal present: Training time is way over estimate
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Deadlock in distributed training
• Data loading is the bottleneck
• Learning rate too small (no visible progress)
• No gradient flow (detached graph)
• All-reduce hang in DDP