DeepSpeed ZeRO-3: Cannot partition a param in flight (save)

save_16bit_model() under ZeRO-3 asserts because parameters are still in flight when the save interval is not a multiple of the gradient-accumulation steps.

DeepSpeed ZeRO-3: Cannot partition a param in flight (save) is a Reliability failure seen during ML training runs. save_16bit_model() under ZeRO-3 asserts because parameters are still in flight when the save interval is not a multiple of the gradient-accumulation steps. Common tags: Deepspeed, Zero 3, Checkpoint, Inflight Param.

Save_16bit_model() lacked the save_checkpoint_prologue() call that save_checkpoint() has. Prologue partitions all parameters before saving. Without it, some params remain INFLIGHT and the partition assert fires. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Checkpoint save crashes under ZeRO-3
• Happens in save_fp16_model / save_16bit_model
• Save interval not aligned to grad accumulation

• DeepSpeed ZeRO-3
• Large models (e.g. T5-11B)
• save_interval % grad_accumulation_steps != 0

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: AssertionError: Cannot partition a param in flight
• ✓Verified signal present: Crash in _zero3_consolidated_fp16_state_dict()
• ✓Verified signal present: save_16bit_model in the stack trace
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

DeepSpeed changes when parameters, gradients and optimizer state are created, partitioned, gathered and offloaded. The hub separates ZeRO, memory, checkpoint and pipeline failures by lifecycle phase.

• save_16bit_model() lacked the save_checkpoint_prologue() call that save_checkpoint() has
• Prologue partitions all parameters before saving
• Without it, some params remain INFLIGHT and the partition assert fires