Silent Data Corruption from GPU Hardware Faults Causing Loss Spikes and Model Divergence

Silent data corruption arising from latent GPU hardware defects bypasses ECC and other hardware detection mechanisms, causing incorrect computation results without any error signal. In LLM training, these faults manifest as sudden loss spikes, NaN propagation, or gradual parameter divergence that may go undetected for thousands of steps. Google reported SDC events weekly to biweekly during Gemini training, and Meta attributed 6 unplanned interruptions to SDC during Llama 3 training.

Silent Data Corruption from GPU Hardware Faults Causing Loss Spikes and Model Divergence is a Reliability failure seen during ML training runs. Silent data corruption arising from latent GPU hardware defects bypasses ECC and other hardware detection mechanisms, causing incorrect computation results without any error signal. In LLM training, these faults manifest as sudden loss spikes, NaN propagation, or gradual parameter divergence that may go undetected for thousands of steps. Google reported SDC events weekly to biweekly during Gemini training, and Meta attributed 6 unplanned interruptions to SDC during Llama 3 training. Common tags: Sdc, Silent Data Corruption, Loss Spike, Gpu Hardware.

Silicon manufacturing escapes and aging create marginal hardware paths that intermittently produce wrong arithmetic results. Voltage/frequency scaling during transient power draw pushes timing margins past failure threshold on marginal units. SDC defects are strongly non-uniform: critical datapaths in tensor cores produce catastrophic divergence at moderate error rates while mantissa errors are masked. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training loss suddenly spikes by 5-100x with no corresponding change in input data or hyperparameters
• NaN values appear in gradients and optimizer states after a specific GPU compute kernel execution
• Model converges to different optima when trained on ostensibly identical hardware

• Large-scale LLM training on 1000+ GPU clusters
• BF16/FP8 training where reduced precision narrows guard bands against timing faults
• Long-duration jobs (>1 week) where latent defects activate under sustained thermal/voltage stress

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Spontaneous loss spikes in training curve without correlating input change or LR schedule event
• ✓Verified signal present: Single-rank gradient norm divergence detected via NCCL all-reduce comparison across replicas
• ✓Verified signal present: nvidia-smi reports zero correctable ECC errors yet computation results differ from replicated runs
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Silicon manufacturing escapes and aging create marginal hardware paths that intermittently produce wrong arithmetic results
• Voltage/frequency scaling during transient power draw pushes timing margins past failure threshold on marginal units
• SDC defects are strongly non-uniform: critical datapaths in tensor cores produce catastrophic divergence at moderate error rates while mantissa errors are masked