Loading benchmark results…
ML engineers trust numbers they can reproduce. Denpex's regex and retrieval matchers are benchmarked against 78 unique, class-labeled GPU training incident recordsmined from public GitHub issues, with a CI regression gate that detects a measured drop. These results do not measure the entire diagnostic engine. You don't have to take our word for any number here, scroll down and run the real matcher in your own browser, then recompute the published IDF + anchors row over the actual corpus.
Encyclopedia entries
718
published failure pages
Unique incidents
78
mined from public GitHub issues
Curated (real traces)
55
with an error signature
Regex + IDF full top-1
79.5%62/78; Wilson 95% CI 69.2% to 87.0%; full development corpus; independent incidents
Full development corpus, not the full product
78 unique incident records mined from public GitHub issues across major training frameworks (PyTorch, Lightning, HuggingFace, DeepSpeed, FSDP, vLLM). Each carries a mined failure class; known label noise is disclosed, and reviewed corrections carry a label note. We report two slices:
Three retrieval strategies are compared so you can see exactly what each contributes:
runDiagnosis, governance checks, abstention, or LLM escalation, so it is not a full-product accuracy measurement.The product surfaces up to 3 distinct failure classes(“most likely X, or Y/Z”), not 3 entries of the same class. The eval mirrors this: it pulls a deeper candidate pool and dedupes by class, so top-3 is a real class-recall metric rather than collapsing onto top-1. Accuracy is measured at the canonical class level via an independent taxonomy (see worker/eval/taxonomy.ts).
Reproducible via npm run eval . Or right here in your browser, below. FULL is the default result; CURATED is a secondary slice.
| Matcher | Full top-1 | Full top-3 | Curated top-1 | Curated top-3 |
|---|---|---|---|---|
| naive token-overlap (old) | 64.1%50/78; Wilson 95% CI 53.0% to 73.9%; full development corpus; independent incidents | 76.9%60/78; Wilson 95% CI 66.4% to 84.9%; full development corpus; independent incidents | 72.7%40/55; Wilson 95% CI 59.8% to 82.7%; curated development corpus; independent incidents | 81.8%45/55; Wilson 95% CI 69.7% to 89.8%; curated development corpus; independent incidents |
| IDF + anchors (new) | 71.8%56/78; Wilson 95% CI 61.0% to 80.6%; full development corpus; independent incidents | 78.2%61/78; Wilson 95% CI 67.8% to 85.9%; full development corpus; independent incidents | 81.8%45/55; Wilson 95% CI 69.7% to 89.8%; curated development corpus; independent incidents | 81.8%45/55; Wilson 95% CI 69.7% to 89.8%; curated development corpus; independent incidents |
| regex + IDF cascade | 79.5%62/78; Wilson 95% CI 69.2% to 87.0%; full development corpus; independent incidents | 87.2%68/78; Wilson 95% CI 78.0% to 92.9%; full development corpus; independent incidents | 92.7%51/55; Wilson 95% CI 82.7% to 97.1%; curated development corpus; independent incidents | 94.5%52/55; Wilson 95% CI 85.1% to 98.1%; curated development corpus; independent incidents |
Curated top-1 receipts on the same development corpus are: naive token overlap 40/55 (72.7%; Wilson 95% CI 59.8% to 82.7%; curated development corpus; independent incidents), IDF + anchors 45/55 (81.8%; Wilson 95% CI 69.7% to 89.8%; curated development corpus; independent incidents), and the regex + IDF cascade 51/55 (92.7%; Wilson 95% CI 82.7% to 97.1%; curated development corpus; independent incidents). These are distinct matcher configurations, so the complete receipts replace a relative-lift percentage. The aggregate is affected by label noise and records whose useful context exists only in the issue title rather than the submitted trace.
The fastest way to settle “is this real retrieval or a pile of ifstatements?” is to run it. Paste a failure log and watch the real IDF + anchors retrieval stage rank it, including the exact signatures and tokens that fired. Then recompute that published row over the real corpus. No account, no clone, no data leaving your machine.
Don't take the table's word for it. This downloads the real labeled corpus and re-runs the shipping IDF + anchors retrieval stage over every record, right here. It checks the complete result receipts against the separate IDF + anchors row published above.
Where the matcher is strong, and where the label noise lives. Classes with trustworthy labels (OOM, silent hang, compile, device-assert) are strong; the noisy-label classes drag the aggregate.
VERSION_MISMATCH11 logs · 32 entriesTORCH_COMPILE10 logs · 9 entriesCHECKPOINT_CORRUPTION6 logs · 34 entriesDEVICE_ASSERT6 logs · 12 entriesSILENT_HANG6 logs · 97 entriesFSDP_ERROR6 logs · 13 entriesOOM_FRAGMENTATION5 logs · 60 entriesDEEPSPEED_ERROR4 logs · 33 entriesDDP_ERROR1 logs · 13 entriesThis checks whether the standalone retrieval matcher's confidence predicts its observed class-match rate. It does not calibrate the confidence shown by the complete product path. In an ideal matcher, higher confidence corresponds to a higher observed class-match rate. The mid-band is where this matcher is over-confident and where active work is focused.
| Confidence band | Samples | Observed top-1 class match |
|---|---|---|
| 85 to 100% | 3 | 100.0%3/3; Wilson 95% CI 43.9% to 100.0%; curated development corpus; independent incidents |
| 75 to 84% | 7 | 100.0%7/7; Wilson 95% CI 64.6% to 100.0%; curated development corpus; independent incidents |
| 65 to 74% | 15 | 86.7%13/15; Wilson 95% CI 62.1% to 96.3%; curated development corpus; independent incidents |
| 55 to 64% | 22 | 77.3%17/22; Wilson 95% CI 56.6% to 89.9%; curated development corpus; independent incidents |
| 0 to 54% | 6 | 50.0%3/6; Wilson 95% CI 18.8% to 81.2%; curated development corpus; independent incidents |
Every pull request runs the full eval as a CI gate. A change that drops curated accuracy below the floors (or below the naive baseline) fails the release gate. A second gate asserts the numbers on this page are byte-identical to the eval. Floors are ratcheted upward as the matcher improves. The ratios below are configured release policies, not observed measurements, so sampling intervals do not apply to them. Every measured result above carries its sample count and Wilson interval.
Curated regex + IDF cascade top-1 floor
ratio >= 0.89 · enforced policy
Curated regex + IDF cascade top-3 floor
ratio >= 0.91 · enforced policy
IDF + anchors at least naive baseline (no regression)
top-1 & top-3 · enforced
Clean-label classes (OOM, hang, compile, assert) top-3
ratio >= 0.85 · enforced policy
Every labeled class represented in the encyclopedia
all labeled classes · enforced
This page's numbers match the eval (no drift)
drift-guard · enforced
Honest caveat: the corpus is mined from public GitHub issues, so it skews toward reproducible, well-documented failures - not the long tail of bespoke internal errors. Labels originate in the mining workflow and still contain known noise. The AI tier and live search are not measured here, and the in-browser recompute exercises only the encyclopedia matcher. These numbers therefore measure a narrow, deterministic slice of the product.
The labeled corpus and the exact results above are downloadable. Score them with your own tooling, or diff them against your runs. The corpus is public GitHub issue data; every record carries its source_url.
With repo access, the eval, taxonomy, and regression gates run locally:
npm install npm run eval # prints the full table aboveRead the docs