False Negative Analysis

Definition

A false negative is when the engine should have fired a rule on a fault event but did not.

Measurement

We instrumented the live rules and ran them in shadow for 30 days. Faults that were confirmed by user-side post-mortems but had no denvex-fired event:

• False negative rate: 5.4 % overall.

Causes

1. Missing tracepoints — kprobe into nvidia.ko required but not available on A100/H100 for some Xid events. Fix: NVML fallback + DCGM cross-check.
2. Async anomaly propagation — DBE occurred 6 minutes before the user's job observed it. Fix: lower thresholds; add 90s forward window.
3. Pinned to unexpected GPU — rank-to-host mapping was wrong by one PCIe root complex. Fix: validate via TopologyGraph before fusing into evidence.
4. Per-layer weight explosion — only flagged at layer granularity, missed when global tensor exploded faster. Fix: add a global tensor snapshot at 5s cadence.

Knowledge Gaps

Repro

benchmarks/false-negatives/fault-injection/run.sh is the canonical regression test that drops in known faults and asserts each is caught.