Data Integrity error references

Browse 30 published references for data integrity failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• botocore ClientError 503 PutObject SlowDown S3 prefix request rate exceeded

  S3 throttling (503 SlowDown), request rate exceeded per-prefix limits. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent checkpoint-storage failures.

• Checkpoint Corruption

  Corrupted checkpoints silently poison resumed training with bad weights. Denpex validates checkpoint integrity before resume.

• Checkpoint Saved with Newer Version

  Checkpoints saved with newer PyTorch versions may not load with older versions, causing compatibility issues.

• Checkpoint Saved with Older Version

  Checkpoints saved with older PyTorch versions may not load with newer versions due to format changes.

• Checkpoint Torn Write

  Torn writes produce incomplete checkpoint files when training is interrupted mid-save.

• Checkpoint Version Incompatible

  Checkpoint version incompatibility prevents loading checkpoints saved with different PyTorch versions.

• Dataset License Issue

  Dataset license issues prevent commercial use or distribution of models trained on the data.

• DeepSpeed DecoupledCheckpointEngine Infinite Hang

  DeepSpeed's DecoupledCheckpointEngine can hang indefinitely during checkpoint saving, blocking training progress without producing an error. The hang occurs when the async checkpoint writer thread deadlocks or the checkpoint queue fills up. Denpex detects the hang by monitoring checkpoint save duration.

• DeepSpeed FastFileWriter File Descriptor Leak, Phantom ENOSPC

  DeepSpeed's FastFileWriter leaks file descriptors during checkpoint saves, eventually causing 'No space left on device' (ENOSPC) errors even when disk space is available. The leaked file descriptors exhaust the per-process fd limit, making the filesystem appear full. Denpex detects the fd leak pattern from ENOSPC errors with available disk space.

• DeepSpeed SIGBUS Exit Code -7 After Checkpoint Load

  DeepSpeed training crashes with SIGBUS (exit code -7) after loading a checkpoint, with no Python traceback. This typically indicates a memory-mapped file access failure. The checkpoint file was truncated, deleted, or on a filesystem that was unmounted. Denpex detects the SIGBUS pattern and correlates it with checkpoint file integrity.

• DeepSpeed Universal Checkpoint Lexicographic Sort Corruption

  DeepSpeed universal checkpoint loading sorts rank directories lexicographically instead of numerically, causing rank_10 to be loaded before rank_2. This silently corrupts model weights because each rank's parameters are loaded into the wrong rank's optimizer state. Denpex detects the corrupted checkpoint by validating parameter checksums across ranks.

• DistCheckpointError Async checkpoint saving failed in background thread write timeout

  A background distributed-checkpoint writer missed its storage deadline. The job may continue computing while a rank is blocked on storage, but that checkpoint must not be treated as complete or recoverable. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent checkpoint-storage failures.

• Model State Dict Corruption

  Model state dict corruption produces degraded model quality after loading.

• Optimizer State Dict Mismatch

  Optimizer state doesn't match model parameters when loading checkpoints across different architectures or configs.

• Optimizer State Dict Mismatch

  Optimizer state doesn't match model parameters when loading checkpoints across different architectures or configs.

• Optimizer State Load Failure on Cluster Resize

  ZeRO optimizer states are heavily coupled to the exact partitioning topology (world size) present when the checkpoint was saved. If a model was trained on 8 GPUs, there are 8 sharded optimizer state files. Standard DeepSpeed loading expects a 1:1 mapping and cannot dynamically reshard the Adam states to, say, 16 GPUs.

• Partial Checkpoint Save

  Partial checkpoint saves write incomplete data when training is interrupted, leaving a file that appears valid but is truncated.

• PII Leakage in Training Data

  PII in training data can leak through model outputs, causing privacy and compliance issues.

• PyTorch Load Failed / Unpickling Error

  Checkpoint load fails when files are corrupted or format is mismatched.

• PyTorch Save Failed / Write Error

  Checkpoint save fails when disk writes are interrupted.

• Safetensors Load Error

  Safetensors loading fails when files are truncated or checksums mismatch.

• Silent Data Corruption: Gradual Parameter Drift

  A permanent hardware fault causes monotonically increasing divergence of parameters from a baseline once injection begins, even while loss curves look identical to a healthy run.

• Silent Data Corruption: Silent Degradation (No NaN)

  A faulty GPU corrupts computation without ever producing a NaN. Loss settles slightly above baseline and parameters drift, making it the most deceptive SDC mode.

• Single-Rank Reduce-Scatter Silent Data Corruption

  On world_size=1 with non-aligned tensor sizes, NCCL's reduce-scatter/AVG left the last elements unprocessed (zero) due to a floor-division bug. Silent corruption with no error.

• sudden gradient norm spike without learning rate change transient DRAM parity error

  The gradient norm jumped by orders of magnitude with no change to the schedule. Most such spikes are a data or numerical problem, but a spike with no corresponding loss anomaly, on one rank only, is one of the few observable signatures of silent data corruption. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent sdc failures.

• tensor core GEMM calculation deviation silent data corruption bit flip

  A GEMM result differs from a trusted recomputation without a CUDA exception. That is a silent data corruption signal, but the comparison must first rule out expected floating-point non-determinism and an input mismatch. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent sdc failures.

• TensorRT-LLM refuses a quantized checkpoint because its weight mapper does not recognise the export layout

  TensorRT-LLM loads a checkpoint through a per-model weight mapper that expects tensor names and groupings in a particular layout. A quantized export written by a different tool, or by a different version of the same tool, can be entirely valid and still be rejected, because the mapper has no rule for the names it contains.

• UnpicklingError pickle data was truncated EOFError Ran out of input checkpoint

  Checkpoint File Corruption - The PyTorch checkpoint file is truncated or corrupted, likely due to a previous crash during `torch.save()`. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent checkpoint-storage failures.

• webdataset TarError Unexpected end of archive while reading shard

  A WebDataset tar shard ended mid-record. The shard is truncated, either the upload was incomplete, or an HTTP stream was cut and the reader treated the partial body as the whole file. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent checkpoint-storage failures.

• Weights-only load failed with an UnpicklingError when you resume an older checkpoint

  A checkpoint that loaded without complaint for months starts failing after a framework upgrade. Nothing about the file changed. The loader now refuses by default to reconstruct any Python object that is not a plain tensor, and a training checkpoint carries several, the optimizer class among them, so the load stops on the first one it meets.