Real Trace Walkthroughs
10 anonymized real-trace walkthroughs with timeline screenshots. Each is shipped in case-studies/ and indexed here.
| # | Title | Summary |
|---|---|---|
| 2 | "DBE on H100, training silently degraded for 9 minutes before drop" | NVML ECC counter rose 18s before checkpoint hash mismatch; canary microbatch detected SDC; rerouted. |
| 3 | "NCCL watchdog tripped on rank 7 of 32 after switch reset" | UFM reported SXid 22013; root cause pinned to switch port 17; job rerouted to spare fleet. |
| 4 | "Thermal throttle storm in cluster row B" | PSI plus DCGM throttle counts detected the 5-minute pre-failure trend; drainage prevented full outage. |
| 5 | "Storage OST saturation on shared Lustre" | OST p99 latency rose 4× baseline; identify culprit OST; shifted workload. |
| 6 | "PCIe Gen5 downshift to Gen3 on a single H100" | NVML flag fired before any user-visible symptoms. |
| 7 | "Layer checksum drift on FSDP all-gather boundary" | Per-layer XXH3 caught the silent reordering; masked out the slow rank. |
| 8 | "Megatron TP all-reduce hang" | EBPF NCCL probes pinpointed a single comm primitive pinning to a particular port. |
| 9 | "JAX HLO compile crash on shard 3" | Compile time went from 8s to 64s; rule fired; warm-spare accepted. |
| 10 | "Repeating first-failed-rank on node-12" | First-failed-rank tracker escalated after 2 events in 18h; node drained. |
Each walkthrough includes:
- A timeline screenshot.
- A JSON dump of the CausalOrder.
- A remediation outcome log.
- A retrospective: "how did detection latency compare to customer-side detection?"
Data
JSON dumps and timelines live in case-studies/data/. The visualizations are rendered by tools/timeline-render from a single JSON.