HBM3 Memory Failures on 16384-GPU Cluster During Llama 3 Training

During the 54-day pre-training of Llama 3 405B on 16,384 NVIDIA H100 GPUs, 419 unexpected interruptions occurred at an average rate of one every 3 hours. HBM3 memory failures alone accounted for 17.2% of all unplanned interruptions, while total GPU-related failures (including NVLink issues) comprised 58.7%. Silent data corruption caused 6 additional undetected job interruptions.

HBM3 Memory Failures on 16384-GPU Cluster During Llama 3 Training is a Fail-Slow failure seen during ML training runs. During the 54-day pre-training of Llama 3 405B on 16,384 NVIDIA H100 GPUs, 419 unexpected interruptions occurred at an average rate of one every 3 hours. HBM3 memory failures alone accounted for 17.2% of all unplanned interruptions, while total GPU-related failures (including NVLink issues) comprised 58.7%. Silent data corruption caused 6 additional undetected job interruptions. Common tags: Meta, Llama3, Hbm3, H100.

HBM3 memory modules on H100 GPUs exhibit wear-out under sustained 700W thermal load, transitioning from correctable to uncorrectable ECC errors without warning. Temperature-induced voltage/frequency scaling amplifies timing margin violations in HBM3 I/O. Silent data corruption from latent manufacturing defects bypasses ECC detection and silently poisons gradient computation. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training job interrupted every 2-3 hours on average with no single root cause
• HBM3 memory errors logged as ECC correctable followed by uncorrectable errors
• Model training silently corrupts on SDC-affected nodes with no explicit error signal
• Checkpoint recovery overhead compounds with each additional interruption

• NVIDIA H100 SXM clusters at 1024+ GPU scale
• Long-duration training (>7 days) on multi-tenant shared clusters
• PyTorch FSDP / Megatron-LM with frequent all-reduce collectives

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: dmesg logs showing HBM3 ECC correctable error counts rising over time before uncorrectable errors
• ✓Verified signal present: NCCL watchdog timeouts with nvidia-smi reporting falling GPU memory bandwidth
• ✓Verified signal present: Training loss curve shows repeated sawtooth pattern from checkpoint rewinds
• ✓Verified signal present: Xid 79 GPU has fallen off the bus preceded by thermal and power excursions
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• HBM3 memory modules on H100 GPUs exhibit wear-out under sustained 700W thermal load, transitioning from correctable to uncorrectable ECC errors without warning
• Temperature-induced voltage/frequency scaling amplifies timing margin violations in HBM3 I/O
• Silent data corruption from latent manufacturing defects bypasses ECC detection and silently poisons gradient computation