Gpu Degradation Guide
What We Detect
| Signal | Threshold | Rule |
|---|---|---|
| SBE count / day | > 10 | WARNING |
| DBE | > 0 | CRITICAL (deterministic_root_candidate) |
| Retired pages (SBE) | > 0 | INFO |
| Retired pages (DBE) | > 0 | CRITICAL |
| Row remap pending | > 0 | WARNING |
| Row remap fail | > 0 | CRITICAL |
| Thermal throttle count / hour | >= 1 | WARNING |
| Hw slowdown count / hour | >= 1 | WARNING |
| Power violation | >= 1 | WARNING |
| PCIe replay count / hour | >= 1 | WARNING |
| PCIe link gen | < 4 | WARNING |
| PCIe link width | < 16 | WARNING |
| NVLink replay count / hour | >= 1 | WARNING |
| NVLink CRC flit error | >= 1 | WARNING |
Predictive Engine
The prediction engine observes longer-horizon trends:
- 24h rolling mean of SBE rate.
- 7d rolling mean of throttle count.
- Weibull hazard rate per entity.
- Peer-relative z-score via Welford.
Output: Vec<DegradationForecast>:
HardwareAtRisk— risk score ≥ 0.6.ReplacementRecommended— hazard rate > 2× peer median.SubtleDrift— peer-relative z > 3.TelemetryPipelineDegraded— scrape latency > 2× peer median.
What Action Should You Take?
| Finding | Action |
|---|---|
| SBE/day > 10 | Drain + RMA |
| Throttle rate trending up 3 days | Schedule maintenance |
| PCIe gen < 4 | Replace riser, check cables |
| NVLink replay > 0 | Inspect cable |
| Weibull hazard > 2× peer median | Replace within 30 days |
Automated Remediation
Default policy:
- RunDiagnostic (AutoSafe): runs on every DetectionEvent.
- Drain (Approve): never auto. Requires SRE approval.
- RMA (AutoSafe): always auto-issue.
You can override via policy.yaml:
- action: drain
tier: auto_remediate
conditions:
- kind: tenant
id: acme
- kind: severity
at_least: warningRepro
benchmarks/gpu-degradation/ injects each degradation signal on a controlled GPU and verifies the rule fires.
See Also
docs/hardware-degradation-validation.md— validation methodology.docs/prediction-methodology.md— statistical basis.