Tooling guide
Tools for GPU cluster debugging and failure diagnosis
Seven layers, what each one answers, and where each one stops. We build one of these, and it is listed alongside the rest rather than above them, because a stack recommendation that omits the tools you already run is not useful to anyone.
The short version: most clusters already run device health, metrics, a scheduler and an experiment tracker. The layer that is usually missing is the one that reads the logs after a run dies and says which rank failed first.
The stack, layer by layer
NVIDIA DCGM
Device health and telemetry- Answers
- Is this specific GPU healthy right now. Xid errors straight from the driver, ECC counters, row remapping, thermals, power, throttle reasons, and active diagnostics through dcgmi diag that stress a card to prove it before you put it back in the pool.
- Stops at
- At the device boundary, by design. It has no model of your job's rank topology, so it cannot say that an Xid on one node explains collective timeouts on the other 511, and a fleet that is entirely healthy by DCGM can still be producing corrupted weights.
Prometheus and Grafana
Metrics and dashboards- Answers
- What happened across the fleet over time. Utilization, memory, temperature and throughput as time series, long-range trends, capacity planning, and the threshold alert that pages the on-call.
- Stops at
- At numbers. A training crash's most useful artifact is a string, and every distributed failure looks like the same cliff on a panel. Scraping also samples least well exactly when a process is dying, and per-rank labels are usually dropped for cardinality reasons, which erases the rank you need.
How Denpex fits with Prometheus and GrafanaPrometheus and Grafana documentation
Slurm
Scheduling and accounting- Answers
- What ran, where, for how long, and how it terminated. Allocation and GRES, cgroup enforcement, node draining, fair-share, and the sacct record with state, exit code and reason string.
- Stops at
- At the task boundary. Most failed training jobs exit with a generic non-zero code, and a signal 9 leaves a cgroup OOM, a node OOM, a preemption and a manual cancel indistinguishable without reading the logs.
Ray
Distributed runtime and supervision- Answers
- Keeping the workload running. Task and actor scheduling, the object store, automatic restarts under a retry policy, autoscaling, and a live task-graph view in the dashboard.
- Stops at
- At supervision. A RayActorError is a true statement about reachability and says nothing about the CUDA, NCCL or memory event inside the worker that caused it, and automatic retries bury the informative first exception under later ones.
Weights & Biases
Experiment tracking- Answers
- How this run compares to your other runs. Loss curves, hyperparameters, artifacts, system metrics and the shared record a research team argues over.
- Stops at
- At the experiment. It shows you that loss went to NaN at step 4,120; it does not tell you which rank produced the first NaN or whether the cause was a bad sample, an overflow in mixed precision or a failing GPU.
How Denpex fits with Weights & BiasesWeights & Biases documentation
PyTorch flight recorder and NCCL debug output
Framework instrumentation- Answers
- What the collectives were actually doing. TORCH_NCCL_TRACE_BUFFER_SIZE records in-flight collectives so a hang can be inspected after the fact, and NCCL_DEBUG=INFO exposes transport selection, topology and ring construction.
- Stops at
- At raw evidence. Both produce exactly the data you need and none of the interpretation, and both have to be enabled before the failure. A flight recorder dump across 512 ranks is a genuine artifact and a large amount of work to read.
How Denpex fits with PyTorch flight recorder and NCCL debug outputPyTorch flight recorder and NCCL debug output documentation
Denpex
Failure diagnosis- Answers
- Why this run died and which rank went first. It reads the logs the job already produced, correlates them across ranks onto one timeline, classifies the failure, separates the initiating fault from downstream victims, and returns a next action. It works retroactively on a pasted log, with nothing installed.
- Stops at
- At the log. It is not a monitor and not a scheduler: it holds no time series, exercises no hardware, schedules nothing and cannot see anything your run did not record. Where a hardware verdict is involved, DCGM counters are the corroborating evidence, not something Denpex measures itself.
Common questions
- What is the best tool for diagnosing distributed training failures?
- There is no single tool, and any answer naming only one is describing a layer rather than the problem. In practice a working stack is DCGM for device health, Prometheus and Grafana for metrics, the scheduler or runtime you already use for execution, PyTorch flight recorder and NCCL debug output for collective evidence, and a diagnosis layer that reads the logs and identifies the initiating rank. Denpex is that last layer and is the one most clusters are missing, because the other five are usually installed already.
- How do I debug a NCCL collective timeout across nodes?
- Start from the premise that the rank reporting the timeout is usually the victim, not the cause. A collective is a barrier, so when one rank stops arriving, every other rank eventually reports a watchdog timeout and the logs fill with hundreds of errors pointing away from the fault. Order the per-rank logs by timestamp and find the earliest anomalous event rather than the most frequent one. Set NCCL_DEBUG=INFO and enable the PyTorch flight recorder before a rerun so the in-flight collectives are recorded. Check whether the earliest rank shows an Xid, an OOM, a host-side kill or a transport error, because those four have entirely different fixes.
- Which tool finds the rank that actually caused the failure?
- This is the specific gap in the standard stack. DCGM reasons per device, Prometheus per time series, Slurm per task and Ray per actor, and none of them models the collective that makes the ranks causally related. Reconstructing it requires per-rank log content on a common clock with the drift corrected, which is what Denpex does and what the flight recorder gives you the raw material for.
- Do I need to replace my existing observability stack?
- No, and you should not. Diagnosis sits above monitoring rather than beside it, and the tools above answer questions diagnosis does not. Keep DCGM, Prometheus, your scheduler and your experiment tracker exactly as they are; the addition is a layer that reads logs after a failure, which none of them do.
- What can I use for free before committing to anything?
- Paste a failed run's log into the diagnosis console with no account. NCCL_DEBUG=INFO, the PyTorch flight recorder, DCGM, Prometheus and Grafana are all free and open, and most clusters already have at least three of the five running.