← Back to reference

Correlation Engine Design

Goals

  1. Group low-level events into a single CausalOrder per fault.
  2. Identify the most likely root cause node (GPU, NIC, OST, etc.).
  3. Surface supporting evidence with confidence scores.

Graph Model

CausalOrder
├── nodes: Vec<EvidenceNode>
│    ├── Event { ts, kind, source }
│    ├── Rank { rank, host, job_id }
│    ├── GPU { uuid, host, model, mig_mode }
│    ├── PCIeBus { bus_id, gen, width }
│    ├── SwitchPort { switch, port }
│    ├── OST { ost_id, fs }
│    ├── MDT { mdt_id, fs }
│    ├── NIC { ib_dev, port, host }
│    ├── Node { hostname, rack }
│    └── Rack { rack_id, row }
├── edges: Vec<EvidenceEdge>
│    ├── causes       (X → Y)
│    ├── follows      (X in time after Y)
│    ├── on_same_host
│    ├── same_step
│    └── communicates_with
└── score: Confidence { value, breakdown }

Root-Cause Selection

For each CausalOrder we compute a per-node score:

score(node) = sum(rule_weight(node) for rule in rules(node))
              * source_reliability(source)
              * proximity_factor(ts_delta_seconds)
              + recurrence_bonus(node)
              - conflicting_evidence_penalty(node)

Top-1 by score = "most likely root cause." Output on every DiagnosisEvent.

Time Correlation

Events are bucketed by step window (default 60s). Cross-bucket links require a follow-up event within 5m that references the same host/rank/OST/PCIeBus/nic.

Output

agents/denpex-control-plane/src/correlation/mod.rs produces:

  • CausalOrder — full graph (used for render-graph).
  • RuleHit — list of fired rules with evidence.
  • MostLikelyRootCause — top-1 + alternates.

Tests cover:

  • Single-cause chains.
  • Multi-cause faults.
  • Cascading failures (one root → multiple symptoms).