Checkpoint Integrity Guide

What Denpex Validates

Per checkpoint save:

Manifest hash. The manifest's SHA-256 must match the committed denpex.manifest.commit.manifest_sha256.
Shard hashes. Each shard file's hash must match rank_owners[rank].
LastModified ordering. No shard may be modified after the manifest's commit time.
Smoke inference. A 30-second dummy-model load on a warm spare proves the shards are usable.

What Denpex Adds to the Manifest

{
  "version": 1,
  "format": "denpex.v1",
  "shards": {
    "0": { "path": "shards/rank-0.bin", "size": 12345678, "sha256": "abc..." },
    "1": { "path": "shards/rank-1.bin", "size": 12345679, "sha256": "def..." }
  },
  "commit": {
    "manifest_sha256": "f8e0...",
    "time_unix_nano": 1700000000000000000,
    "rank_owners": { "0": "abc...", "1": "def..." }
  },
  "denpex_meta": {
    "denpex_version": "0.7.0",
    "policy_version": 3,
    "smoke_inference_required": true
  }
}

Hash Algorithms

Algorithm	Use
SHA-256	Standard for the manifest
XXH3-128	Sampled for fast per-shard check

The combination is selectable via denpex_checkpoint_hash env var.

Atomic Commit Marker

We use an atomic commit marker (write-then-rename pattern) to prevent partial writes:

target_path.tmp.<uuid>  ->  target_path

If we observe a .tmp.* left over after the manifest, we raise StorageEvent::PartialWriteDetected.

On Restart

After process restart:

Load the manifest.
Validate SHA-256 of every shard.
Optionally run smoke inference.
Resume.

On Failure

CHECKPOINT_MANIFEST_MISMATCH — abort restore.
CHECKPOINT_SHARD_HASH_MISMATCH — quarantine the affected shard.
CHECKPOINT_SMOKE_INFERENCE_FAIL — escalate the job.

Compatibility

Works with PyTorch native save, FSDP SHARDED state dict, DeepSpeed ZeRO-3 checkpoint format, Megatron distributed checkpoint.