Denpex vs Slurm
Slurm runs the job and records that it failed. Denpex explains why. The scheduler and the diagnosis are not the same layer.
What is Slurm?
Slurm is the dominant workload manager on HPC and GPU clusters. It handles queueing and fair-share scheduling, allocates nodes and generic resources such as GPUs, launches tasks through srun and slurmstepd, enforces cgroup limits, tracks node health and drains bad nodes, and records the accounting record you read back with sacct. It is the system of record for what ran, where, for how long, and how it terminated.
What is Denpex?
Denpex is a failure diagnosis engine for distributed training and inference runs. It reads the output the job itself produced, correlates it across ranks onto one timeline, classifies the failure against 16,400+ known signatures, and returns a ranked root cause with its supporting evidence and a specific next step. It sits above the scheduler and does not care which scheduler you use.
The core difference
Slurm knows about tasks; your training job knows about ranks; the failure happens in the second world and is reported in the first. When a 64-node job dies, sacct gives you a job-level state and slurmstepd may report a signal on one task. What it cannot tell you is that task 17's process hit an uncorrectable ECC error, that the remaining 63 nodes were blocked in an all-reduce waiting for it, and that every one of their watchdog timeouts is a consequence rather than a separate fault. Slurm records the termination faithfully. It has no model of the collective that made those terminations related.
Feature comparison
| Capability | Denpex | Slurm |
|---|---|---|
| Schedules jobs, allocates nodes and GRES | No | Yes |
| Enforces cgroup memory and CPU limits | No | Yes |
| Records job state, exit code and elapsed time (sacct) | No | Yes |
| Drains and reboots unhealthy nodes | No | Yes |
| Fair-share accounting and QOS | No | Yes |
| Explains why exit code 1 happened | Yes | No |
| Separates an OOM kill from a NCCL timeout from a hardware fault | Yes | Signal only |
| Identifies which rank inside the allocation failed first | Yes | No |
| 11,700+ failure patterns with prescriptive fixes | Yes | No |
| Distinguishes a preemption from a real crash | Yes | Reports the reason string |
| Works on Kubernetes, Ray and bare torchrun as well | Yes | No |
Verdict
Slurm is your scheduler and it stays your scheduler. It owns allocation, GRES, cgroups, node health and accounting, and nothing about diagnosis changes any of that. What Slurm gives you at the end of a failed run is a state, a signal and a reason string. Denpex reads the logs that job produced and turnsFAILED, exit code 1 into a named failure, an initiating rank and a next action.
Key differences explained
Exit code 1 is not a diagnosis
The overwhelming majority of failed training jobs terminate with a generic non-zero exit, because that is what Python does when a process raises. A signal 9 narrows it to a kill and still leaves the cgroup OOM killer, the node OOM killer, a scheduler preemption and a manual scancel indistinguishable without reading further. Denpex reads further, and returns the specific class rather than the exit status.
The scheduler sees tasks, the failure happens between ranks
Slurm's unit is the task and its view stops at the process boundary. NCCL failures are relational: they are about what rank A was waiting for from rank B. Reconstructing that needs per-rank log content on a common clock, which is precisely the artifact Denpex consumes and the one sacct has no reason to hold.
Preemption and crash look alike downstream
On a preemptible or shared partition, a job that lost its allocation and a job that crashed both leave a pile of collective errors on the surviving ranks. Slurm records the preemption in the job reason, which is authoritative when you read it. Teams frequently do not, because the first thing they see is a wall of NCCL output. Denpex reads both the scheduler output and the rank logs together, so a preemption is reported as a preemption instead of being investigated as a fabric fault.
One diagnosis across schedulers
Most teams do not run only Slurm. There is a Slurm cluster, a Kubernetes cluster, some Ray, and researchers running torchrun by hand on a dev box. Denpex works from logs, so the same diagnosis applies across all of them and the answer does not change shape when the scheduler does. That is not something a scheduler-specific tool can offer.
When to use each
Use Slurm when…
You need scheduling, fair-share, GRES allocation, cgroup enforcement, node draining, reservations or accounting. You want the authoritative record of what ran and how it terminated. You are managing cluster utilization and queue policy.
Use Denpex when…
sacct says FAILED and you need to know why before requeueing. You want the initiating rank rather than the loudest error. You need to tell a hardware fault from a config error from an application bug, so the job goes to the right team and the retry actually has a chance of succeeding.
Use both when…
Standard on an HPC or AI cluster. Slurm runs the job and records the outcome; Denpex reads the logs from that job and explains the outcome. The Slurm reason string and slurmstepd output are useful evidence in the diagnosis, so the two reinforce each other rather than competing.
Integration path
No migration and no changes to your Slurm configuration. Denpex reads logs, so the simplest starting point is pasting a failed job's output into the free diagnosis tool. If you install the agent it wraps the training command inside your existing sbatch script and does not alter the scheduler, the cgroup configuration, prolog and epilog scripts, or node health checks.
Frequently asked questions
Does Denpex replace Slurm?
No, and there is no overlap to replace. Slurm schedules and accounts for work. Denpex diagnoses a run that already failed. You need a scheduler either way, and Denpex has no scheduling functionality at all.
Can sacct tell me why my job failed?
It tells you the state, the exit code, the signal and the reason string, which is genuinely the right starting point and often enough for an obvious case such as a timeout or a preemption. For a distributed training crash it usually reports a generic non-zero exit on a job whose logs contain hundreds of errors, and the cause is in the logs rather than in the accounting record.
How do I tell a cgroup OOM from a CUDA OOM?
They are different failures with different fixes and they look similar from the scheduler's side. A cgroup or node OOM kill shows as a signal 9 with kernel OOM-killer output on the host and means host RAM, usually the dataloader. A CUDA OOM is a Python exception about device memory and means the model or batch does not fit in HBM. Denpex separates them from the log evidence and names which one happened.
Does Denpex parse slurmstepd output?
Scheduler output included in the logs you submit is treated as evidence like any other signal, and slurmstepd lines are frequently decisive for distinguishing a kill from a crash. There is no live Slurm API integration or sacct reader today, so this works from the logs you provide rather than by querying the controller.
We use Slurm and Kubernetes. Do we need two tools?
Not for diagnosis. Denpex works from logs, so the same engine and the same answers apply on both, as well as on Ray and on bare torchrun. The scheduler stays whatever it is on each cluster.