Control Plane Architecture

← Back to reference

The control plane is the correlation/diagnosis/prediction/retrieval layer that ingests telemetry from the host agent and GPU collector streams, applies deterministic rules, and emits remediation actions with confidence scores.

Components

+-------------------+      +-------------------------+
| OTLP receiver     | ---> | Stream-by-stream        |
+-------------------+      | validation + enrichment |
                          +-------------------------+
                                       |
                                       v
+-------------------+      +-------------------------+
| Causality graph   | <--> | TopologyGraph (K8s+K8s) |
+-------------------+      +-------------------------+
         |
         v
+-------------------+
| Deterministic     |  -> RuleHit { rule_id, evidence, confidence }
| rule engine (7+)  |
+-------------------+
         |
         v
+-------------------+
| Confidence        |  -> Confidence { value, breakdown }
| scorer            |
+-------------------+
         |
         v
+-------------------+
| RAG retrieval     |  -> Top-K (Incident, score)
| (lancedb/candle)  |
+-------------------+
         |
         v
+-------------------------+
| Remediation dispatcher  | -> Slack/PagerDuty/Jira + state machine
+-------------------------+

Deterministic Rules

The control plane ships with seven deterministic rules:

ID	Trigger	Severity	Required Evidence
R2	DBE > 0	CRITICAL	DCGM_FI_DEV_ECC_DBE_VOL_TOTAL
R3	NCCL async error	CRITICAL	NCCL_DEBUG=INFO log, AllReduce timeout
R4	NVLink replay > 0 (hourly rate)	WARNING	DCGM NVLink counter
R5	PCIe link gen < 4	WARNING	NVML max/current PCIe gen
R6	Storage latency spike	WARNING	OST jobstats rolling 1m
R7	Hardware detachment (NVML persistent fault)	CRITICAL	NVML flag violation

Each rule is a pure function over Evidence slices. Output is RuleHit { rule_id, evidence_ids, severity }.

Confidence Scoring

score_diagnosis(evidence: &[Evidence]) -> Confidence combines:

Deterministic-rule weight (50%) — number of independent rules that fired.
Source reliability (20%) — Xid > DCGM > log > user.
Timestamp proximity (15%) — events within 30s window get full credit.
Recurrence count (10%) — same rule firing N times in 24h.
Conflicting-evidence penalty — apply subtraction.
Missing-data penalty — apply subtraction.

Result: Confidence { value: 0.0..1.0, breakdown: BTreeMap } — must appear on every RuleHit and every CausalOrder.

Retrieval Engine

Local vector store backed by lance (lancedb equivalent for embedded use) with embeddings produced by a quantized sentence-transformer via candle-transformers. On every DiagnosisEvent we embed "{rule_id} {root_cause} {evidence_summary}" and return the top-K most similar past incidents. Each result includes the supporting rule treatise.

Evidence Graph

Every diagnosis emits a JSON graph (GraphML-compatible) with:

Nodes: Event, Rank, GPU, PCIeBus, SwitchPort, OST, MDT, NIC, Node, Rack.
Edges: causes, follows, on_same_host, same_step, communicates_with.

CLI: denpex-control-plane render-graph <report.json> --out graph.json.

Prediction Engine

prediction/feature_store.rs (SQLite-backed EntityHistory):

Rolling windows of SBE/thermal/row-remap/PCIe-gen/width per entity.
Rules-based risk score (weighted sum).
Weibull hazard rate per entity.
Peer-relative z-score via Welford.
Observability-aware: if scrape latency for a GPU is > 2× peer median and the last 10 samples had gaps > 30s, raises Prediction::TelemetryPipelineDegraded with high confidence.

Output: Vec<DegradationForecast> enriched with predicted_failure_ns, confidence, human_reason.