False Positive Analysis
Definition
A false positive is when the engine fires a deterministic rule on a non-fault event. We track:
- False-positive rate = (FP / total non-fault observations).
- FP burst rate = average FP count per cluster-hour.
Measurement
From 30 days of shadow-mode evaluation on production-equivalent clusters (anonymized):
| Cluster tier | FP rate | FP per cluster-hour |
|---|---|---|
| 64 GPU | 0.6 % | 0.16 |
| 1024 GPU | 0.9 % | 0.51 |
| 4096 GPU | 1.2 % | 1.42 |
Top Causes
- Threshold sensitivity —
R1 (Xid)fires on Xid 62 (training interrupts that are benign in some frameworks). Fix: replace ≥ 1 with = in allow-list; default to allow-list {48, 63, 64, 79, 94}. - Counter resets —
SBE>0fired on counter wrap-around that lasted seconds. Fix: rate of change instead of absolute. - NCCL log noise —
NCCL async errorrule fired onWARNlines that were recoverable. Fix: require ERROR level.
Drift Suppression
docs/operations/drift-suppression.md describes the routing of suspected FPs into a "shadow" bucket so users can inspect them.
How to Report a False Positive
Send the incident id + a short note to our support team. We push a rule adjustment within one release cycle.