1-2 GPU Hardware Failures Per Week During BLOOM 176B Training on 384 A100 GPUs

During the 4-month training of the 176B-parameter BLOOM model on 384 NVIDIA A100 GPUs, the team experienced 1-2 GPU hardware failures per week on average. Each failure required a 30-60 minute node replacement and checkpoint rollback. The cluster used 48 nodes with 32 spare GPUs available for hot-swap replacement.

Symptom

XID 79 (GPU fallen off bus) or XID 48 (double bit ECC error) in dmesg

Root cause

Infant mortality of GPU components (HBM2e, NVLink bridges) in newly provisioned cluster. Thermal cycling during training start/stop wears solder joints on GPU package. Single GPU failure brings down its entire 8-GPU node, affecting all parallel ranks on that host.

Recommended fix

kubectl cordon <node>

How Denpex helps

Denpex investigates 1-2 GPU Hardware Failures Per Week During BLOOM 176B Training on 384 A100 GPUs using the evidence you provide or your connected workload collects. Earlier rank, host or application evidence is needed to distinguish an initiating failure from a downstream report.

1-2 GPU Hardware Failures Per Week During BLOOM 176B Training on 384 A100 GPUs is a Reliability failure seen during ML training runs. During the 4-month training of the 176B-parameter BLOOM model on 384 NVIDIA A100 GPUs, the team experienced 1-2 GPU hardware failures per week on average. Each failure required a 30-60 minute node replacement and checkpoint rollback. The cluster used 48 nodes with 32 spare GPUs available for hot-swap replacement. Common tags: Huggingface, Bloom, A100, Hardware Failure.

Infant mortality of GPU components (HBM2e, NVLink bridges) in newly provisioned cluster. Thermal cycling during training start/stop wears solder joints on GPU package. Single GPU failure brings down its entire 8-GPU node, affecting all parallel ranks on that host. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training interrupted 1-2 times per week by GPU failure detected via XID error or NCCL timeout
• Failed GPU replacement required 30-60 minutes even with hot spares available
• Checkpoint rollback loses 1.5 hours of training progress on average

• NVIDIA A100 80GB clusters with 384+ GPUs
• Long-duration training runs (>1 month) on research clusters
• New cluster deployments where hardware infant mortality is high

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: XID 79 (GPU fallen off bus) or XID 48 (double bit ECC error) in dmesg
• ✓Verified signal present: NCCL timeout error on specific rank that maps to a single GPU on a single node
• ✓Verified signal present: nvidia-smi shows the affected GPU as ERR! with non-zero retired pages count
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

• Infant mortality of GPU components (HBM2e, NVLink bridges) in newly provisioned cluster
• Thermal cycling during training start/stop wears solder joints on GPU package
• Single GPU failure brings down its entire 8-GPU node, affecting all parallel ranks on that host

XID 79 (GPU fallen off bus) or XID 48 (double bit ECC error) in dmesg
nvidia-smi shows the affected GPU as ERR! with non-zero retired pages count

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.