Gray Failure Detection Methodology
What Counts as a "Gray Failure"
A gray failure is hardware or software that is partially working: it returns a result, but the result is degraded in a way that lowers MTTR if caught early. Examples:
- Straggler ranks running 10× slower than peer median.
- NVLink replay storm that's silently eating bandwidth.
- Thermal throttle without SP alert.
- PCIe link gen downshift that "still works" but at half the bandwidth.
Detection Primitives
| Detector | Math | Threshold |
|---|---|---|
| NCCL collective latency drift | per-op z-score vs. 24h rolling | z >= 3.0 |
| GPU bandwidth drift | per-gpu.uuid rolling z-score | z >= 3.0 |
| Thermal throttle count | events/hour | >= 1 |
| PCIe replay count | events/hour | >= 1 |
| Storage latency spike | OST p99 vs. 1m baseline | >= 4× |
| Repeated first-failed rank | per node / 24h | >= 2 within 18h |
Implementation
- Welford's algorithm for streaming mean + variance per (job, rank).
- BTreeMap-keyed rolling windows with explicit capacity.
- Per-entity feature store (SQLite) for cross-job drift detection.
Tests
- Synthesized gray-failure scenarios injected into the feature store, verified detection.
- Weight anomaly tests using simulated layer-norm streams.
Delivery
Each detected gray failure becomes a GrayFailureDetection trigger for remediation. The default tier is AutoSafe (drain probes) — actual remediation still requires an approval unless the rule explicitly maps it higher.
Coverage Gaps
- Network gray failures inside RDMA NIC firmware are not observable from userspace; we rely on counter trends.
- We do not detect gray failures in user-supplied kernels — only in the vendor + libc + framework calls.