← Back to reference

Prediction Engine Design

Why

Failures rarely "appear" — they creep. By the time a deterministic rule fires, the hardware has often been degrading for days. The prediction engine's role is to surface those gradients early enough that the user can schedule proactive maintenance rather than reactive eviction.

Inputs

For each entity (GPU.uuid, NIC.dev:port, OST.uuid, MDT.uuid, switch port, host), the feature store keeps a rolling window of:

  • sbe_total (lifetime + rolling 1h, 24h, 7d)
  • dbl_bit_count
  • retired_pages_*
  • row_remap_pending / row_remap_fail
  • thermal_throttle_count
  • hw_slowdown_count
  • pcie_replay_count
  • pcie_curr_link_gen / pcie_curr_link_width
  • power_violation_count

Plus a per-entity scrape latency / sample-gap histogram for observability-aware degradation.

Algorithms

Rules-Based Risk Score

risk(entity) =
    0.30 * sbe_rate_today
  + 0.25 * throttle_rate_today
  + 0.20 * remap_rate
  + 0.15 * hw_slowdown_rate
  + 0.10 * pcie_replay_rate

Score is normalized to 0..1, and risk >= 0.6 raises a Prediction::HardwareAtRisk with confidence = 0.7.

Survival Analysis (Weibull)

For each entity we fit a Weibull distribution (shape k, scale λ) to the time-to-failure history across the fleet. The hazard rate at age t is h(t) = (k/λ) (t/λ)^(k-1). Entities with h(age) > fleet median × 2 get a Prediction::ReplacementRecommended.

Peer-Relative Anomaly (Welford)

Per metric, we maintain a running mean + variance via Welford's algorithm across the fleet. A new sample with z > 3 raises a Prediction::SubtleDrift.

Telemetry-Pipeline Health

If scrape latency for a GPU is > 2× its peer median AND the last 10 samples had gaps > 30s, raise Prediction::TelemetryPipelineDegraded { gpu_uuid, confidence }.

Outputs

Vec<DegradationForecast> with:

pub struct DegradationForecast {
    pub entity_id: String,
    pub kind: ForecastKind,
    pub predicted_failure_ns: u128,
    pub confidence: f64,
    pub human_reason: String,
}

The control plane serializes each DiagnosisEvent with the forecasts attached, and the AI layer renders a natural-language summary.

Synthetic Tests

  • Synthetic time-series producing known hazard rates.
  • Peer-relative z-score against known distributions.
  • Telemetry-pipeline degradation with simulated scrape gaps.