The ML failure diagnosis platform
A deterministic pipeline that collects logs from all ranks, correlates failure events using clock-drift corrected ordering, classifies them against 11,700+ documented patterns, and outputs an evidence-ranked root cause and concrete next action.
How the diagnosis pipeline works
Six sequential steps from process exit to prescriptive next action. Known signatures follow the fast deterministic path; novel and ambiguous failures intentionally spend longer testing competing hypotheses.
Multi-rank log collection
The Denpex agent wraps your training command and captures its stdout and stderr. Cross-rank diagnosis requires logs from the affected ranks through your launcher or scheduler integration. Collection coverage and latency depend on that configuration; large multi-host fleets require a scoped acceptance test.
Clock-drift correction
Clock skew can obscure the order of distributed failures. Denpex correlates available timestamps and supplied clock-offset evidence with rank exceptions and collective progress. Missing timing evidence limits causal conclusions and may require another diagnostic capture.
Deterministic matching and evidence retrieval
The corrected log corpus is matched against Denpex's pattern library: 11,700+ documented failure patterns covering NCCL, CUDA, PyTorch, DeepSpeed, FSDP, Megatron-LM, and GPU hardware faults. Each pattern includes the root cause, the affected systems, and a prescriptive next action. JAX/Flax and XLA stack traces are accepted today via the deeper fallback; native JAX pattern packs are in progress on the roadmap.
Failure classification
Evidence-supported classifications cover hardware, numerical, data pipeline, environment and distributed communication failures. Routing follows your configured alert channels and policies. Missing or conflicting evidence can leave the initiating cause unresolved.
Prescriptive fix output
The alert summarizes the observed failure and recommended next step, with a link to the full evidence and verification plan. A recommendation is not a verified repair. Checkpoints, configuration changes and node isolation require workload-specific safety checks; insufficient evidence prompts a targeted diagnostic request.
AI analysis for unfamiliar failures
When no deterministic pattern matches with sufficient confidence, Denpex routes the available, policy-permitted evidence to a deeper diagnosis path. Ambiguous incidents can take several minutes and may ask for a discriminating artifact instead of inventing a root cause. Novel failures are flagged for review before deterministic coverage is added.
Coverage counts describe different units: 10,800+ deterministic matcher rules, 700+ published encyclopedia entries, and 4,700+ community incidents. The combined evidence corpus contains 16,400+ signatures, including 11,700+ documented patterns. These counts do not measure distinct failure classes or diagnosis accuracy. See the benchmark methodology and measured retrieval results.
How Denpex turns failure evidence into a next action
Paste your logs. Get a diagnosis. Fix the problem.
Paste your logs
Copy the error output from your training job. Works with PyTorch DDP, FSDP, DeepSpeed ZeRO, Megatron-LM, and Axolotl. Paste it into the diagnosis box.
Check known failure signals
Denpex applies deterministic rules to your logs, then checks supporting evidence and competing explanations. A known error signal can support a diagnosis; ambiguous cases need deeper analysis or another diagnostic check.
16,400+ failure signatures covered
CUDA OOM, memory fragmentation, NCCL timeout, gradient explosion, checkpoint corruption, import errors, version mismatches, device asserts, silent hangs, and more. 11,700+ are documented patterns that ship with a prescriptive fix; the rest are real-world incidents and their resolutions, reached by semantic recall. Measured regex + IDF cascade top-1 63/78 (80.8%; Wilson 95% CI 70.7% to 88.0%; full development corpus; independent incidents) and top-3 66/78 (84.6%; Wilson 95% CI 75.0% to 91.0%; full development corpus; independent incidents) across a retrieval index of 721 published encyclopedia entries. This is a matcher benchmark, not full-product accuracy. Index size is coverage, not accuracy.
AI analysis for unfamiliar failures
If your failure doesn't match a known pattern, Denpex uses AI to analyze and suggest what happened. Always get a next step, even for novel errors.
This error pattern doesn't match known failures. AI analysis suggests checking memory allocator configuration and batch size settings.
One line. No config. Works on your next failure.
import denpex
# Add before your training loop
denpex.init(
api_key="dpx_...",
job_name="llama3-70b-finetune",
notify=["slack", "sms"] # optional
)
# The rest of your training code is unchanged
trainer.train()Monitoring tells you the job died. Denpex tells you why.
W&B and Grafana are dashboards. ChatGPT has no idea what rank 42 was doing. Denpex is the only tool built specifically to diagnose distributed GPU training failures, and hand you the fix.
Diagnose from paste-only logs
Don't need an agent or integration. Paste your error output, get a diagnosis. Works with any framework. PyTorch, DeepSpeed, Megatron, Axolotl, JAX/Flax, whatever you're using.
Prescriptive fixes, not just error codes
Denpex doesn't just tell you what broke. When the evidence supports a verdict, it returns a specific env var, config change, containment step, or checkpoint action. When evidence is missing, it asks for the safest discriminator instead of guessing.
10,800+ deterministic matcher rules
Rules cover CUDA, NCCL, framework, and hardware signals. A rule match identifies a known signal; supporting evidence determines whether it explains the incident. Semantic recall reaches 16,400+ total failure signatures, including 4,700+ real-world incidents. Coverage counts are not accuracy measurements. Measured regex + IDF cascade top-1 63/78 (80.8%; Wilson 95% CI 70.7% to 88.0%; full development corpus; independent incidents) and top-3 66/78 (84.6%; Wilson 95% CI 75.0% to 91.0%; full development corpus; independent incidents) across a retrieval index of 721 published encyclopedia entries. This is a matcher benchmark, not full-product accuracy. Index size is coverage, not accuracy.
Instant diagnosis, no waiting
Pattern matching runs in seconds without generative AI on the deterministic path. Novel or ambiguous errors use deeper analysis with confidence and evidence controls, and can ask for more signal instead of forcing a verdict.
Goodput and fleet reliability
Calculate tokens/sec, point-in-time MFU, achieved PFLOP/s, wasted GPU-equivalent compute, node risk scores, and RMA candidates from the same run and fleet evidence operators already collect.
Preflight, scheduler actions, and RMA packets
Run supported topology preflight checks, generate qualified SLURM/Kubernetes runbooks, and produce RMA evidence packets with device details and verification steps. A hardware replacement decision requires confirmation; remote execution requires separate pilot qualification and action approval.
| Capability | Manual debugging | W&B / Grafana | Generic LLM chat | Denpex |
|---|---|---|---|---|
| Attributes the initiating rank when evidence supports it | hours of log diving | No | No no cluster context | Yes evidence-backed or honestly unknown |
| Root cause, not symptom dashboards | Eventually | No shows metrics, not causes | Guesses | Yes deterministic signals with evidence checks |
| Concrete next action: env var, config, checkpoint, or evidence check | If you find it | No | Unverified suggestions | Yes prescriptive when safe; asks when uncertain |
| Remembers your cluster's failure history | Tribal knowledge | No | No | Yes team knowledge base |
| Works at 2am with zero setup | No you are the setup | No needs instrumentation | Yes | Yes paste logs, done |
Platform FAQ
How does Denpex handle clock skew between nodes?
Denpex uses available timestamps and clock-offset evidence alongside rank exceptions and collective sequence information. A timestamp alone does not prove which rank initiated a failure. When timing evidence is incomplete, the investigation should request a discriminating capture rather than promise exact ordering.
What is the maximum cluster size Denpex supports?
Denpex is designed for and supports clusters up to 16,384 GPUs. Synthetic tests exercise cascade analysis and correlated-event coalescing at large scale. They do not establish real multi-host GPU-fabric recovery, log coverage or notification latency. Qualify your actual launcher, topology and alert workflow in a scoped pilot before expanding deployment.
Does Denpex require changes to my training code?
The job wrapper captures process output without modifying training code. Cross-rank collection and optional numerical or layer telemetry can require launcher, scheduler or training instrumentation. Verify the evidence collected from one workload before expanding.
How does silent data corruption detection work?
When configured instrumentation supplies numerical and per-layer telemetry, Denpex can flag anomalous update distributions for investigation. An anomaly is not proof of silent data corruption. Reproduce the affected computation and compare it with a trusted result before attributing a hardware cause.
Can Denpex run entirely within my VPC?
The on-host deterministic engine supports DENPEX_LOCAL=1 to block Denpex control-plane requests. Install the offline engine first and verify egress controls in your environment. Agent strict mode sends minimized failure signatures over the network and is separate from local operation. Hosted dashboards, cloud fallback and hosted notifications are separate services and are not part of zero-egress local diagnosis. In-VPC deployments require qualification.
How do I verify installation?
Download the agent or install the published SDK, configure a scoped API key, and wrap one controlled training command. Confirm a heartbeat, failure evidence, an accepted test alert and the same incident report. The quickstart covers SLURM, Kubernetes and bare metal; installation time depends on your environment.