DeepSpeed DecoupledCheckpointEngine Infinite Hang

DeepSpeed's DecoupledCheckpointEngine can hang indefinitely during checkpoint saving, blocking training progress without producing an error. The hang occurs when the async checkpoint writer thread deadlocks or the checkpoint queue fills up. Denpex detects the hang by monitoring checkpoint save duration.

DeepSpeed DecoupledCheckpointEngine Infinite Hang is a Data Integrity failure seen during ML training runs. DeepSpeed's DecoupledCheckpointEngine can hang indefinitely during checkpoint saving, blocking training progress without producing an error. The hang occurs when the async checkpoint writer thread deadlocks or the checkpoint queue fills up. Denpex detects the hang by monitoring checkpoint save duration. Common tags: Deepspeed, Checkpoint, Hang, Deadlock.

The async checkpoint writer thread deadlocks when the checkpoint queue is full and the main thread tries to enqueue another checkpoint. NVMe offload checkpoint writes to slow storage (NFS) can block the writer thread indefinitely. A race condition in the DecoupledCheckpointEngine causes the writer thread to wait for a signal that is never sent. Memory pressure causes the checkpoint buffer allocation to block, which in turn blocks the writer thread. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training hangs indefinitely during checkpoint save with no error output
• The checkpoint save operation never completes, blocking all training progress
• GPU utilization drops to 0% during the hang while the process remains alive

• DeepSpeed ZeRO-3 with DecoupledCheckpointEngine
• DeepSpeed with NVMe offload checkpointing
• Training with frequent checkpoint saves and slow storage (NFS, S3)
• DeepSpeed versions before v0.14.2

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Training stops making progress at a checkpoint save step
• ✓Verified signal present: nvidia-smi shows 0% GPU utilization on all ranks
• ✓Verified signal present: DeepSpeed logs show 'Saving checkpoint...' but no completion message
• ✓Verified signal present: ps aux shows the checkpoint writer thread stuck in D (uninterruptible sleep) state
• ✓Verified signal present: No timeout or error is triggered. The job just hangs forever
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

DeepSpeed changes when parameters, gradients and optimizer state are created, partitioned, gathered and offloaded. The hub separates ZeRO, memory, checkpoint and pipeline failures by lifecycle phase.

• The async checkpoint writer thread deadlocks when the checkpoint queue is full and the main thread tries to enqueue another checkpoint
• NVMe offload checkpoint writes to slow storage (NFS) can block the writer thread indefinitely
• A race condition in the DecoupledCheckpointEngine causes the writer thread to wait for a signal that is never sent
• Memory pressure causes the checkpoint buffer allocation to block, which in turn blocks the writer thread