Cross-Run Environment Diff
“Your last good run was Tuesday 3pm. Since then: NVIDIA driver 550→570, PyTorch 2.4→2.6, and the NCCL config changed.” This tool extracts both environments, diffs them, and ranks every change by how strongly it correlates with your failure — so you bisect the right thing first instead of third.
Frequently asked questions
What do I paste in?
Whatever you have from each run: pip freeze or conda list, the nvidia-smi header (driver + CUDA), NCCL version lines from logs, env dumps, container image tags, uname -r. The extractor takes one messy blob per run — order doesn't matter, and missing pieces just narrow the diff.
What does the correlation percentage mean?
It is a transparent heuristic, not causal proof: each change is weighted by (a) how often that component class co-occurs with the observed failure class in the Denpex incident corpus, (b) the magnitude of the change (major > minor > patch), and (c) +15 points when the failing output literally names the component. The weights are printed with every row so you can disagree with the ranking at a glance. Its job is to tell you where to bisect FIRST.
Does this replace bisection?
It replaces the days of guessing which of the 14 things that changed deserves the first bisect. When the top suspect is 90%+ (e.g. a driver major bump against an Xid failure), reverting that one component usually settles it in a single run.
Want this automatic?
Team+ plans get the same engine at POST /api/diagnose/env-diff, and the agent snapshots every run's environment so the diff against your last good run is attached to the diagnosis before you even ask.