Denpex vs Framework error messages
An error message is a symptom. Denpex reads every rank's logs and tells you the root cause.
What is Framework error messages?
Framework error messages are the diagnostic output built into PyTorch, DeepSpeed, FSDP, Megatron-LM, NCCL, and CUDA. When a training job fails, these frameworks write error strings to stderr, typically the last exception seen by each rank before it exited. They are designed to be human-readable and to report what the process observed at the moment it died. They are not designed to identify root causes, correlate across ranks, or distinguish between the originating fault and the cascade it triggered.
What is Denpex?
Denpex is a purpose-built ML training failure diagnosis engine. It correlates available rank logs, applies clock-drift correction to build a causal timeline, classifies the failure against 11,700+ documented failure patterns, and returns an evidence-ranked root cause, originating rank, and prescriptive next action. It treats framework error messages as one input signal among many, not as the diagnosis itself.
The core difference
Framework error messages report what each process observed at the moment it died. Denpex correlates the evidence available across the distributed system. The difference is the difference between a symptom and a diagnosis. 'NCCL timeout on rank 0' tells you rank 0 stopped receiving responses. When the logs contain the causal chain, Denpex can show that rank 42 experienced an Xid 48 first, followed by the watchdog cascade; when they do not, it reports identity as unknown and asks for the missing evidence.
Feature comparison
| Capability | Denpex | Framework error messages |
|---|---|---|
| Stops at the first error message seen | . | Yes |
| Cross-rank correlation: causal order across all ranks | Yes | No |
| Identifies the initiating rank when the evidence supports attribution | Yes | No |
| Classifies as hardware / ML / data / environment | Yes | No |
| Resolves ambiguous errors: 'NCCL timeout' → actual fault | Yes | No |
| Returns a concrete action or the safest missing-evidence check | Yes | No |
| AI fallback for novel errors | Yes | No |
| Routes alert to the right engineer with ownership mapping | Yes | No |
| Available without installing anything extra | No (one pip install) | Free, built into the framework |
| Pricing | From $0 (3 free diagnoses); Team $415/month billed annually ($499 month-to-month) | Free (the framework) |
Verdict
Framework error messagesare symptoms, and they are frequently wrong. “NCCL timeout” is the watchdog's last guess, not the root cause. “CUDA error: device-side assert triggered” is the consequence of a bug, not its location. Denpexreads every rank's log stream, clock-drift-corrects them into causal order, and tells you the originating fault. Usually a different rank and a different failure class than the framework reported.
Key differences explained
NCCL timeout is almost never what actually failed
The most common error message in distributed training is 'NCCL watchdog timeout.' It can appear across ranks when collective communication stalls, but it is usually a cascade symptom rather than the initiating cause. Triggers include a GPU fault, a network interface failure, a straggler, an application deadlock, or an OOM. Denpex correlates the rank logs that were actually supplied, identifies the likely initiator only when the evidence supports it, and otherwise names the exact missing discriminator.
The reporting rank is rarely the failing rank
In a distributed job, the rank that writes the most alarming error message is typically not the rank that originated the failure. Rank 0 is often the last to die because it's monitoring other ranks. The error you see on rank 0 is the error from the watchdog, not the error from the actual fault. Denpex identifies the originating rank by building a clock-drift-corrected causal timeline and finding the earliest divergence point: the rank whose behavior changed first.
'CUDA error: device-side assert triggered' tells you nothing
'CUDA error: device-side assert triggered' is one of the most frustrating error messages in PyTorch. It indicates a CUDA kernel assertion failed, but it doesn't tell you which assertion, in which kernel, with which input, on which tensor. It often appears hundreds of steps after the actual bug was introduced. Denpex correlates the CUDA error with the per-layer weight-delta trace to identify when the anomaly first appeared, which layer produced it, and whether the root cause is a data pipeline issue, a numerical precision issue, or a hardware fault.
Framework messages can't detect silent failures
Framework error messages only exist when a process terminates or raises an exception. Silent data corruption (SDC), stragglers, and gray failures produce no framework error messages. Training continues to completion while producing corrupted weights or running at significantly reduced throughput. These failure modes are invisible to any system that relies on framework error output. Denpex detects them through hardware telemetry and statistical analysis of the training signal.
When to use each
Use Framework error messages when…
For a quick sanity check before investigating further. For understanding the immediate exception that caused a process to exit. For simple single-GPU failures where the error message is unambiguous.
Use Denpex when…
Any distributed training failure involving more than one rank. Any NCCL timeout, collective communication hang, or watchdog error. Any failure where the reported error seems inconsistent with the actual problem. Any situation where you need the originating rank, failure class, and fix rather than just the last error message.
Use both when…
Always. Framework error messages are the raw input. Denpex is the diagnostic layer that interprets them, correlates them across ranks, and produces an actionable output. You cannot replace one with the other because they operate at different levels of the stack.
Integration path
No migration. Denpex wraps your existing training command with a single-line change (python denpex.py run -- your_training_script.py). It intercepts stdout, stderr, and hardware telemetry at the process level. Your framework, your training code, and your error messages all remain completely unchanged. Denpex adds a diagnosis layer on top, not inside.
Frequently asked questions
If NCCL timeout appears on all ranks, which rank actually failed?
The watchdog lines alone do not prove an initiating rank. Denpex builds a clock-corrected timeline from the available rank evidence and attributes the initiator only when an earlier rank-local fault or behavioral divergence supports it. If that evidence is absent, the correct answer is unknown and the diagnosis requests the missing rank logs or flight-recorder evidence.
Why does 'CUDA error: device-side assert triggered' not tell me where the bug is?
CUDA kernel assertions run asynchronously on the device. When an assertion fails, CUDA does not immediately raise an exception on the host. The failure is buffered until the next CUDA synchronization point, which can be hundreds of steps later. By then the context of the original failure is gone. Denpex correlates the timing of the CUDA error with the per-layer weight-delta trace to identify the earliest anomaly point.
Can Denpex diagnose single-GPU training failures?
Yes, though the benefit is smaller. Single-GPU failures produce unambiguous error messages in most cases. Denpex's primary value is in distributed training where cross-rank causality is the hard problem. For single-GPU training, Denpex still classifies the failure, suggests the fix, and detects SDC and hardware faults that would otherwise be missed.
Does Denpex require changes to my training code?
No. Denpex wraps your training command at the process level. You change one line in your launcher script (or sbatch job script) to call denpex.py run. Your training code, your framework calls, and your existing logging are completely unchanged.
What frameworks does Denpex support?
Denpex is framework-agnostic. It works with PyTorch (including FSDP, DDP, and DeepSpeed), TensorFlow, JAX, and any training framework that runs on Linux with NVIDIA GPUs. Framework-specific pattern libraries exist for PyTorch/NCCL failures, CUDA errors, and DeepSpeed ZeRO optimizer failures.
What is the difference between a failure class and an error message?
An error message is the text a process writes when it dies. A failure class is a named, documented pattern that represents a specific root cause: with a known mechanism, a known set of symptoms, and a known fix. For example, 'NCCL watchdog timeout' is an error message. 'NCCL cascade from Xid 48 hardware fault' is a failure class. Denpex maps error messages to failure classes.
How does Denpex handle completely novel failures not in the library?
When no deterministic pattern matches with sufficient confidence, Denpex routes the failure to an LLM-powered diagnosis path. The LLM receives the full cross-rank log corpus, the clock-drift-corrected causal timeline, and hardware telemetryand returns a best-effort diagnosis within 60 seconds. Novel failures are flagged for addition to the deterministic library.
Can Denpex tell me if a failure was caused by a data pipeline bug vs a hardware fault?
Yes. Failure classification is one of Denpex's primary outputs. Every diagnosis returns a classification: hardware, ML/numerical, data pipeline, environment/configuration, or distributed communication. Hardware faults route to the infrastructure team; ML failures route to the training team; data bugs route to the data team. The classification is based on the failure class, not heuristics.