Checkpoint Partial Save

Partial checkpoint save happens when training crashes or is killed before all model state is saved.

Checkpoint Partial Save is a Reliability failure seen during ML training runs. Partial checkpoint save happens when training crashes or is killed before all model state is saved. Common tags: Checkpoint, Partial Save, Reliability, Atomic Write.

Process killed during checkpoint save. Storage full mid-write. Distributed rank failure during save. No atomic write pattern. Pickle can't handle complex objects. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Checkpoint is missing some keys
• Loading checkpoint gives Missing key error
• Model partially recovered after crash

• Long training with crash recovery
• Distributed training with rank failures
• Storage-full or process-killed during save

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Unexpected key(s) in state_dict
• ✓Verified signal present: Missing key(s) in state_dict
• ✓Verified signal present: Checkpoint file size is smaller than expected
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Process killed during checkpoint save
• Storage full mid-write
• Distributed rank failure during save
• No atomic write pattern
• Pickle can't handle complex objects