ML engineers trust numbers they can reproduce. Denpex's diagnostic engine is benchmarked against 171 real, human-labeled GPU training failure logsmined from public GitHub issues, with a CI regression gate that blocks any change that drops accuracy. You don't have to take our word for any number here — scroll down and run the real matcher in your own browser, then recompute the headline figure over the actual corpus.
Encyclopedia classes
441
documented failure types
Real failure logs
171
mined from GitHub issues
Curated (real traces)
85
with an error signature
Top-1 (curated, new)
44.7%
up from 28.2% baseline
171 failure records mined from public GitHub issues across major training frameworks (PyTorch, Lightning, HuggingFace, DeepSpeed, FSDP, vLLM). Each is human-labeled with a failure class. We report two slices:
Three retrieval strategies are compared so you can see exactly what each contributes:
The product surfaces up to 3 distinct failure classes(“most likely X, or Y/Z”), not 3 entries of the same class. The eval mirrors this: it pulls a deeper candidate pool and dedupes by class, so top-3 is a real class-recall metric rather than collapsing onto top-1. Accuracy is measured at the canonical class level via an independent taxonomy (see worker/eval/taxonomy.ts).
Reproducible via npm run eval — or right here in your browser, below. The CURATED row (real error logs) is the headline.
| Matcher | Full top-1 | Full top-3 | Curated top-1 | Curated top-3 |
|---|---|---|---|---|
| naive token-overlap (old) | 31.6% | 50.3% | 28.2% | 41.2% |
| IDF + anchors (new) | 38.6% | 46.2% | 44.7% | 44.7% |
| full pipeline (regex→new) | 45.0% | 52.6% | 57.6% | 57.6% |
The new matcher raised curated top-1 from 28.2% → 44.7% (a 59% relative lift) by anchoring on error signatures instead of raw token frequency. The aggregate is held down by label noise in the mined corpus — several GRADIENT/DDP/VERSION records are GitHub prose mislabeled by the auto-labeler, not the matcher's fault. The per-class breakdown below shows where the matcher is strong.
The fastest way to settle “is this real retrieval or a pile of ifstatements?” is to run it. Paste a failure log and watch the actual shipping matcher rank it — including the exact signatures and tokens that fired — then recompute the headline number over the real corpus. No account, no clone, no data leaving your machine.
Don't take the table's word for it. This downloads the real labeled corpus and re-runs the shipping IDF + anchors matcher over every record, right here — then checks the result against the number published above.
Where the matcher is strong, and where the label noise lives. Classes with trustworthy labels (OOM, silent hang, compile, device-assert) are strong; the noisy-label classes drag the aggregate.
OOM_FRAGMENTATION19 logs · 29 entriesTORCH_COMPILE15 logs · 6 entriesGRADIENT_EXPLOSION11 logs · 4 entriesDEVICE_ASSERT11 logs · 5 entriesCHECKPOINT_CORRUPTION8 logs · 26 entriesVERSION_MISMATCH8 logs · 17 entriesDDP_ERROR7 logs · 9 entriesSILENT_HANG6 logs · 37 entriesDoes the confidence score the product shows actually predict accuracy? In an ideal system, higher confidence = higher observed accuracy. The mid-band is where the matcher is over-confident and where active work is focused — we publish it rather than hide it.
| Confidence band | Samples | Observed top-1 acc |
|---|---|---|
| 85–100% | 2 | 0.0% |
| 75–84% | 8 | 100.0% |
| 65–74% | 8 | 25.0% |
| 55–64% | 42 | 36.0% |
| 0–54% | 13 | 85.0% |
Every pull request runs the full eval as a CI gate. A change that drops curated accuracy below the floors (or below the naive baseline) fails CI and cannot merge. A second gate asserts the numbers on this page are byte-identical to the eval, so the page can never drift from the code. The floors are raised — never lowered — as the matcher improves.
Curated full-pipeline top-1 floor
≥ 38% · enforced
Curated full-pipeline top-3 floor
≥ 42% · enforced
New matcher ≥ naive baseline (no regression)
top-1 & top-3 · enforced
Clean-label classes (OOM, hang, compile, assert) top-3
≥ 60% · enforced
Every labeled class represented in the encyclopedia
100% coverage · enforced
This page's numbers match the eval (no drift)
drift-guard · enforced
Honest caveat:the corpus is mined from public GitHub issues, so it skews toward reproducible, well-documented failures — not the long tail of bespoke internal errors. The AI-fallback tier (LLM + live GitHub/StackOverflow search) handles novel errors the encyclopedia hasn't seen, and is notmeasured here (and not covered by the in-browser recompute, which exercises the encyclopedia matcher only). We publish what we can measure reproducibly; we don't inflate it.
The labeled corpus and the exact results above are downloadable — score them with your own tooling, or diff them against your runs. The corpus is public GitHub issue data; every record carries its source_url.
With repo access, the eval, taxonomy, and regression gates run locally:
npm install npm run eval # prints the full table aboveRead the docs