← Back to reference

Prediction Methodology

Goal

Predict hardware failures before they cause a job failure.

Inputs

Per-entity (GPU.uuid, NIC.dev:port, OST.uuid, MDT.uuid, switch port, host) the feature store keeps:

  • sbe_total (lifetime + rolling 1h, 24h, 7d)
  • dbl_bit_count
  • retired_pages_*
  • row_remap_pending / row_remap_fail
  • thermal_throttle_count
  • hw_slowdown_count
  • pcie_replay_count
  • pcie_curr_link_gen / pcie_curr_link_width
  • power_violation_count
  • Per-entity scrape latency + sample-gap histogram

Output

Vec<DegradationForecast> with:

pub struct DegradationForecast {
    pub entity_id: String,
    pub kind: ForecastKind,
    pub predicted_failure_ns: u128,
    pub confidence: f64,
    pub human_reason: String,
}

pub enum ForecastKind {
    HardwareAtRisk,
    ReplacementRecommended,
    SubtleDrift,
    TelemetryPipelineDegraded,
}

Algorithms

Rules-based risk score

risk = (0.30 * sbe_rate_today
      + 0.25 * throttle_rate_today
      + 0.20 * remap_rate
      + 0.15 * hw_slowdown_rate
      + 0.10 * pcie_replay_rate)

risk >= 0.6 -> HardwareAtRisk (confidence = 0.7).

Weibull hazard rate

Fit (shape, scale) per entity from time-to-failure history. h(t) = (k/λ)(t/λ)^(k-1). Above 2× fleet median -> ReplacementRecommended.

Peer-relative z-score (Welford)

Per metric, fleet-relative mean + variance via Welford. z > 3 -> SubtleDrift.

Telemetry-pipeline degradation

Scrape latency for a GPU > 2× its peer median AND last 10 samples had gaps > 30s -> TelemetryPipelineDegraded.

Validation

Synthetic time-series producing known hazard rates are used in prediction/tests.rs. Coverage spans:

  • Constant-fault injected on day 1 with a known hazard.
  • Step-change in PCB temperature.
  • Counter reset to non-zero.

Limitations

See docs/silent-corruption-detection-limits.md for the related discussion on SDC limits.