Timeline Reconstruction Examples

The CausalOrder builder takes a stream of Evidence and produces a directed graph of event dependencies. These examples show the shape of output and how to read it.

Example 1: A clean Xid 79

{
  "final_state": "Detected",
  "events": [
    { "ts_unix_nano": 1700000000000000000, "kind": "GpuXid", "value": { "xid": 79, "gpu_uuid": "GPU-abc" } },
    { "ts_unix_nano": 1700000000700000000, "kind": "RankTimeout", "value": { "rank": 7, "op": "all_reduce" } },
    { "ts_unix_nano": 1700000000300000000, "kind": "DcgmDiagnostic", "value": { "level": 3, "verdict": "FAIL" } }
  ],
  "edges": [
    { "from": 0, "to": 1, "kind": "causes" },
    { "from": 0, "to": 2, "kind": "causes" }
  ]
}

Reading top-down: the Xid at 0s is the root; the rank timeout at 0.7s is downstream; the diag result confirms the Xid.

Example 2: A cascading failure

This shows a case where one root (PCIe gen downshift) cascades through multiple ranks.

{
  "events": [
    { "ts_unix_nano": 100, "kind": "PcieLinkDownshift", "value": { "gpu_uuid": "GPU-1", "from": 5, "to": 3 } },
    { "ts_unix_nano": 200, "kind": "RankTimeout", "value": { "rank": 0 } },
    { "ts_unix_nano": 210, "kind": "RankTimeout", "value": { "rank": 1 } },
    { "ts_unix_nano": 250, "kind": "RankTimeout", "value": { "rank": 2 } }
  ],
  "edges": [
    { "from": 0, "to": 1 }, { "from": 0, "to": 2 }, { "from": 0, "to": 3 }
  ]
}

The "root" is event 0; the three rank timeouts are siblings under it.

Example 3: Multi-cause

Two distinct roots, both contributing to one observed symptom:

{
  "events": [
    { "ts_unix_nano": 100, "kind": "DbeSpike", "value": { "count": 1 } },
    { "ts_unix_nano": 110, "kind": "NcclWatchdog", "value": { "rank": 5 } },
    { "ts_unix_nano": 150, "kind": "StorageLatencySpike", "value": { "ost": 7 } }
  ],
  "edges": [
    { "from": 0, "to": 1 }, { "from": 2, "to": 1 }
  ]
}

Reading the graph: the DBE drove the NCCL watchdog; an unrelated storage spike did not cause the symptom here.

Rendering

denpex-control-plane render-graph <report.json> --out graph.json produces a D3-friendly JSON.

CLI:

$ denpex-control-plane render-graph incident.json --out graph.json
$ cat graph.json | jq '.nodes | length'