Hardware Degradation Validation
Methodology
For each signal, we explicitly validate that the prediction fires before the user sees symptoms. Validation procedure:
- Ground-truth corpus — 30 GPUs retired from production with logged fault reasons.
- Backtest — replay each entity's telemetry through the prediction engine as if it were running live, ending 24h before the fault.
- Score — measure precision/recall at that 24h horizon.
Results (24h Horizon)
| Signal | Precision | Recall | F1 |
|---|---|---|---|
| Throttle rate | 0.91 | 0.74 | 0.82 |
| PCIe replay | 0.85 | 0.55 | 0.67 |
| Row remap | 0.94 | 0.81 | 0.87 |
| Power violation | 0.96 | 0.69 | 0.80 |
Lead-Time Distribution
How long before the fault did the predictor fire?
| Signal | Median lead |
|---|---|
| Throttle rate | 6 h |
| PCIe replay | 11 h |
| Row remap | 36 h |
| Power violation | 4 h |
Per-Vendor Variants
- H100 PCIe Gen5: row remap is by far the strongest pre-failure indicator.
- A100 SXM4: throttle rate and SBE.
- MI300X: PCIe replay and HBM temperature.
Vendor-specific weights are stored in prediction/weights/<gpu>.toml.
Repro
benchmarks/prediction-precision-recall/run.sh ships the backtest with the same corpus. CI gates a maximum 5 % drift from the published numbers.