A100, H100 and H200 cannot be ranked by one failure-rate number from the studies collected here. Different fleets, workloads and observation periods limit the comparison. This page separates published field studies (what large clusters actually measured, with sources), and the documented failure surface per GPU family, interconnect and provider stack from a snapshot of 559 classified encyclopedia signatures. This snapshot is not a count of the current live encyclopedia or measured Denpex recovery performance.
Published third-party reliability measurements, reported as-published with sources. Denpex reprints these; it did not produce them.
16,384× NVIDIA H100 80GB (SXM), Meta production training · 54-day Llama-3 405B pre-training run
1,056 GPUs (A100 + H100), 11.7M GPU-hours over 2.5 years · 2.5 years of production operation
Oak Ridge Summit, 27,648× V100 · multi-year production study
Multi-cluster A100 study · multi-year
What we deliberately do NOT publish yet: a Denpex-fleet MTBF table. This page does not provide validated Denpex fleet exposure and failure denominators. A measured MTBF claim requires a defined event, observation period and eligible fleet, not an elapsed calendar date or a count of diagnostic signatures.
Counts of DISTINCT documented failure signatures in the public Denpex Failure Encyclopedia (559 published classes). These measure how many documented, fixable ways each slice of the stack fails, a coverage/complexity map for procurement diligence, not a field failure rate. Every count links back to entries any engineer can read in the Failure Encyclopedia - this is auditable, not vibes. 61 of 559 signatures are hardware-class; 13 NVIDIA Xid codes are covered (13, 31, 43, 48, 61, 62, 63, 64, 69, 74, 79, 94, 119).
How many DISTINCT documented ways training fails on/around each GPU family, a complexity map for diligence, not a field failure rate.
| Slice | Documented failure signatures | Critical-severity | Where they concentrate |
|---|---|---|---|
| H100 (SXM/PCIe) | 10 | 3 | Reliability (2) · Hardware (2) · Communication (2) |
| A100 | 4 | 1 | Reliability (2) · Hardware (2) |
| H200 | 2 | 1 | Reliability (1) · Communication (1) |
| B200 / GB200 | 2 | 1 | Reliability (1) · Communication (1) |
NVLink/NVSwitch vs InfiniBand vs RoCE vs EFA vs PCIe, where the fabric bites.
| Slice | Documented failure signatures | Critical-severity | Where they concentrate |
|---|---|---|---|
| PCIe | 22 | 8 | Hardware (14) · Communication (2) · Memory (1) |
| NVLink / NVSwitch | 18 | 6 | Hardware (8) · Communication (4) · Fail-Slow (1) |
| InfiniBand | 17 | 3 | Communication (10) · Network (2) · Distributed Communication (1) |
| RoCE / Ethernet | 7 | 2 | Communication (4) · Network (2) · Hardware (1) |
| AWS EFA | 3 | 1 | Communication (3) |
Failure signatures that reference each provider's stack (EFA, GKE, AKS, Slurm/on-prem...). Higher ≠ worse cloud, it can mean more usage and better-documented failure modes. Read with the studies above.
| Slice | Documented failure signatures | Critical-severity | Where they concentrate |
|---|---|---|---|
| Kubernetes (any cloud) | 10 | 0 | Infrastructure (5) · Communication (2) · Reliability (1) |
| On-prem / HPC (Slurm etc.) | 8 | 1 | Infrastructure (4) · Distributed Training (1) · Environment (1) |
| AWS | 7 | 1 | Communication (3) · Infrastructure (2) · Network (1) |
| Azure | 3 | 0 | Infrastructure (3) |
| Google Cloud | 2 | 0 | Infrastructure (1) · Network (1) |
| CoreWeave | 2 | 0 | Reliability (1) · Infrastructure (1) |
| RunPod | 2 | 0 | Infrastructure (2) |
| Lambda | 1 | 1 | Infrastructure (1) |
Machine-readable JSON (free, no key): GET https://api.denpex.com/api/reliability-index. Cite it in your procurement memo; the sources above carry the field numbers.