Diagnosis Engine Design
Modules
agents/denpex-control-plane/src/diagnosis/confidence.rs— rule-weight + source-reliability scoring.agents/denpex-control-plane/src/diagnosis/gray_failure.rs— z-score, Welford, throttle counters, first-failed-rank tracker.agents/denpex-control-plane/src/diagnosis/weight_anomaly.rs— per-layer weight L2/grad L2/NaN/checksum/optimizer asymmetry.agents/denpex-control-plane/src/diagnosis/nccl_topology.rs— Ring/Tree/CollNet identification + rank-link-down localization.
Deterministic-First Principle
We never emit a DiagnosisEvent until we have at least one RuleHit. If the rule coverage is < 90 % on the labeled corpus the AI layer is the fallback for the human-readable summary only. The root cause is always the top-1 deterministic finding ranked by confidence.
Gray-Failure Engine
| Detector | Input | Threshold |
|---|---|---|
| NCCL collective latency | per-op | z >= 3.0 vs. 24h rolling |
| GPU bandwidth | per-gpu.uuid | z >= 3.0 |
| Thermal throttle count | events/h | >= 1 |
| PCIe replay count | events/h | >= 1 |
| Storage latency spike | OST p99 / 1m | >= 4× baseline |
| Repeated first-failed rank | per node / 7 d | >= 2 within 24h window |
All detectors are deterministic and unit-tested with synthesized time-series.
Per-Layer Anomaly Engine
Inputs come from denpex_sdk.instrument.LayerInstrument which records per step (sampled 1 in 10):
weight_l2,weight_linfgrad_l2,grad_linfactivation_l2nan_count,inf_countchecksum(XXH3-128 of the layer's weight tensor)
Detectors:
WeightExplosion— abs deviation > 3σ OR abs_dev_ratio > 10 % vs. constant baseline.GradientExplosion— same threshold.NaNInfDetected— immediate (no history required).ChecksumDrift— XXH3 changed between adjacent steps.OptimizerAsymmetry— cross-rank m_mean range > threshold.
Synthetic Tests
- Synthesized gray-failure scenarios injected into the feature store.
- Weight anomaly tests with simulated layer-norm streams.
- Optimizer asymmetry with two ranks diverging.