Denpex vs Framework error messages
An error message is a symptom. Denpex reads every rank's logs and tells you the root cause.
What is Framework error messages?
Framework error messages are the diagnostic output built into PyTorch, DeepSpeed, FSDP, Megatron-LM, NCCL, and CUDA. When a training job fails, these frameworks write error strings to stderr, typically the last exception seen by each rank before it exited. They are designed to be human-readable and to report what the process observed at the moment it died. They are not designed to identify root causes, correlate across ranks, or distinguish between the originating fault and the cascade it triggered.
What is Denpex?
Denpex is a purpose-built ML training failure diagnosis engine. It collects logs from all GPU ranks simultaneously, applies clock-drift correction to build a true causal timeline, classifies the failure against 3500+ documented failure patterns, and returns the root cause, the originating rank, the failure class, and the prescriptive fix in under 12 seconds. It treats framework error messages as one input signal among many, not as the diagnosis itself.
The core difference
Framework error messages report what each process observed at the moment it died. Denpex determines what actually happened across the full distributed system. The difference is the difference between a symptom and a diagnosis. 'NCCL timeout on rank 0' tells you rank 0 stopped receiving responses from other ranks. Denpex tells you rank 42 experienced an Xid 48 hardware fault at step 8,143, its watchdog fired, and the failure cascaded to all other ranks 400 milliseconds later, giving you the originating rank, the failure class, and the remediation command.
Feature comparison
Verdict
Framework error messagesare symptoms, and they are frequently wrong. “NCCL timeout” is the watchdog's last guess, not the root cause. “CUDA error: device-side assert triggered” is the consequence of a bug, not its location. Denpexreads every rank's log stream, clock-drift-corrects them into causal order, and tells you the originating fault. Usually a different rank and a different failure class than the framework reported.
Key differences explained
NCCL timeout is almost never what actually failed
The most common error message in distributed training is 'NCCL watchdog timeout.' It appears on every rank when collective communication stalls. But NCCL timeout is a cascade symptom, not a root cause. The actual failures that trigger it include: a GPU hardware fault on one rank (Xid errors), a network interface failure on one node, a straggler rank that's running too slowly, a deadlock in application code, or an OOM on a rank that caused its process to pause. Without cross-rank causal ordering, you cannot distinguish between these cases. Denpex reads all ranks' logs, identifies which rank stopped participating first, and tells you what that rank logged before the cascade began.
The reporting rank is rarely the failing rank
In a distributed job, the rank that writes the most alarming error message is typically not the rank that originated the failure. Rank 0 is often the last to die because it's monitoring other ranks. The error you see on rank 0 is the error from the watchdog, not the error from the actual fault. Denpex identifies the originating rank by building a clock-drift-corrected causal timeline and finding the earliest divergence point: the rank whose behavior changed first.
'CUDA error: device-side assert triggered' tells you nothing
'CUDA error: device-side assert triggered' is one of the most frustrating error messages in PyTorch. It indicates a CUDA kernel assertion failed, but it doesn't tell you which assertion, in which kernel, with which input, on which tensor. It often appears hundreds of steps after the actual bug was introduced. Denpex correlates the CUDA error with the per-layer weight-delta trace to identify when the anomaly first appeared, which layer produced it, and whether the root cause is a data pipeline issue, a numerical precision issue, or a hardware fault.
Framework messages can't detect silent failures
Framework error messages only exist when a process terminates or raises an exception. Silent data corruption (SDC), stragglers, and gray failures produce no framework error messages. Training continues to completion while producing corrupted weights or running at significantly reduced throughput. These failure modes are invisible to any system that relies on framework error output. Denpex detects them through hardware telemetry and statistical analysis of the training signal.
When to use each
Use Framework error messages when…
For a quick sanity check before investigating further. For understanding the immediate exception that caused a process to exit. For simple single-GPU failures where the error message is unambiguous.
Use Denpex when…
Any distributed training failure involving more than one rank. Any NCCL timeout, collective communication hang, or watchdog error. Any failure where the reported error seems inconsistent with the actual problem. Any situation where you need the originating rank, failure class, and fix rather than just the last error message.
Use both when…
Always. Framework error messages are the raw input. Denpex is the diagnostic layer that interprets them, correlates them across ranks, and produces an actionable output. You cannot replace one with the other because they operate at different levels of the stack.
Integration path
No migration. Denpex wraps your existing training command with a single-line change (python denpex.py run -- your_training_script.py). It intercepts stdout, stderr, and hardware telemetry at the process level. Your framework, your training code, and your error messages all remain completely unchanged. Denpex adds a diagnosis layer on top, not inside.
Frequently asked questions
If NCCL timeout appears on all ranks, which rank actually failed?
NCCL timeout is a cascade symptom. It appears on all ranks when one rank stops participating in collective communication. The originating rank is the one whose failure triggered the cascade. Without cross-rank causal analysis, you cannot determine which rank that was. Denpex determines it automatically by building a clock-drift-corrected timeline and identifying the earliest divergence point.
Why does 'CUDA error: device-side assert triggered' not tell me where the bug is?
CUDA kernel assertions run asynchronously on the device. When an assertion fails, CUDA does not immediately raise an exception on the host. The failure is buffered until the next CUDA synchronization point, which can be hundreds of steps later. By then the context of the original failure is gone. Denpex correlates the timing of the CUDA error with the per-layer weight-delta trace to identify the earliest anomaly point.
Can Denpex diagnose single-GPU training failures?
Yes, though the benefit is smaller. Single-GPU failures produce unambiguous error messages in most cases. Denpex's primary value is in distributed training where cross-rank causality is the hard problem. For single-GPU training, Denpex still classifies the failure, suggests the fix, and detects SDC and hardware faults that would otherwise be missed.
Does Denpex require changes to my training code?
No. Denpex wraps your training command at the process level. You change one line in your launcher script (or sbatch job script) to call denpex.py run. Your training code, your framework calls, and your existing logging are completely unchanged.
What frameworks does Denpex support?
Denpex is framework-agnostic. It works with PyTorch (including FSDP, DDP, and DeepSpeed), TensorFlow, JAX, and any training framework that runs on Linux with NVIDIA GPUs. Framework-specific pattern libraries exist for PyTorch/NCCL failures, CUDA errors, and DeepSpeed ZeRO optimizer failures.
What is the difference between a failure class and an error message?
An error message is the text a process writes when it dies. A failure class is a named, documented pattern that represents a specific root cause: with a known mechanism, a known set of symptoms, and a known fix. For example, 'NCCL watchdog timeout' is an error message. 'NCCL cascade from Xid 48 hardware fault' is a failure class. Denpex maps error messages to failure classes.
How does Denpex handle completely novel failures not in the library?
When no deterministic pattern matches with sufficient confidence, Denpex routes the failure to an LLM-powered diagnosis path. The LLM receives the full cross-rank log corpus, the clock-drift-corrected causal timeline, and hardware telemetryand returns a best-effort diagnosis within 60 seconds. Novel failures are flagged for addition to the deterministic library.
Can Denpex tell me if a failure was caused by a data pipeline bug vs a hardware fault?
Yes. Failure classification is one of Denpex's primary outputs. Every diagnosis returns a classification: hardware, ML/numerical, data pipeline, environment/configuration, or distributed communication. Hardware faults route to the infrastructure team; ML failures route to the training team; data bugs route to the data team. The classification is based on the failure class, not heuristics.