A Slurm GPU job can fail before allocation, while the step is being created, inside the workload, or during cleanup. The state decides where to look: pending and rejected jobs are scheduler or request problems, step-creation errors point to node and cgroup setup, and jobs killed after launch require accounting plus application logs.
Start with scontrol show job and sacct, then inspect the node and GRES view the scheduler actually used. CUDA_VISIBLE_DEVICES is rewritten for each job step, so device ordinals inside the allocation are intentionally different from the host-wide numbering an operator sees outside it.
Do not return a drained GPU node to service because a retry happened to start elsewhere. Preserve the drain reason, NVIDIA Xid or ECC evidence, health-check output and device UUID, then prove whether the failure follows the GPU, slot, node or workload before clearing state.
The literal string is what you paste into a search bar, so it is the heading. “Class” is who is at fault in practice, not what the message says.
slurmstepd: error: JOB CANCELLED DUE TO OOM-KILLERThe step exceeded its enforced memory boundary and the kernel killed one or more processes. This message normally refers to host RAM in the Slurm cgroup, not CUDA device memory.
First action: Run sacct -j <jobid> --format=JobID,State,ExitCode,ReqMem,MaxRSS,MaxVMSize and compare MaxRSS with the effective --mem or --mem-per-gpu request.
Full entry: root cause, fix and preventionslurmstepd: task/cgroup: unable to allocate requested memory for GPU stepslurmstepd could not establish the requested memory boundary before the workload started. The request can exceed node RealMemory, or the node cgroup hierarchy and Slurm configuration can disagree.
First action: Run scontrol show job <jobid> and scontrol show node <node>, compare the request with RealMemory, then inspect cgroup.conf and the node cgroup version.
Full entry: root cause, fix and preventionslurmstepd: Job <jobid> exceeded its time limitSlurm ended the allocation after its requested or partition-imposed wall time elapsed. The scheduler behaved as configured, but the workload did not checkpoint and exit within the available signal window.
First action: Run scontrol show job <jobid> to compare TimeLimit and RunTime, then set --signal=B:TERM@<seconds> and prove the job writes a valid checkpoint before the limit.
Full entry: root cause, fix and preventionsbatch: error: Batch job submission failed: Requested node configuration is not availableNo node in the selected partition can satisfy the combined CPU, memory, GPU type, count and feature constraints. A value can be valid by itself while the complete resource shape is impossible.
First action: Run sinfo -o "%P %G %c %m %f %t" and compare every --gres, --constraint, --mem, task and CPU requirement as one request rather than testing fields separately.
Full entry: root cause, fix and preventionsqueue shows job in PD state with Reason=ReqNodeNotAvail or Nodes required for job are DOWN, DRAINED or reservedEvery node that satisfies the job constraints is unavailable, drained, down or reserved. More waiting does not make an impossible constraint fit, and retrying the same request does not change node state.
First action: Run squeue -j <jobid> -o "%.18i %.2t %.30R" and sinfo -R, then inspect each candidate with scontrol show node <node> before changing the request.
Full entry: root cause, fix and preventionsbatch: error: QOSMaxSubmitJobPerUserLimit or QOSMaxGRESPerUserThe requested job or aggregate user allocation exceeds a Quality of Service or association limit. Free GPUs can exist while policy still prevents this user or account from consuming them.
First action: Run sacctmgr show assoc user=$USER format=Cluster,Account,QOS,GrpTRES,MaxTRES and squeue -u $USER, then reduce the request or wait for existing usage to clear.
Full entry: root cause, fix and preventionNode state DRAIN: NVML health check timed out GPU uncorrectable ECCThe node health check could not complete while the GPU also carried uncorrectable ECC evidence. This is a node-health incident and must not be cleared by a blind scheduler resume.
First action: Keep the node drained, run nvidia-smi -q and DCGM diagnostics out of band, and preserve dmesg plus the GPU UUID before any reset or power cycle.
Full entry: root cause, fix and preventionNCCL hangs during initialization when torchrun is launched from an srun task per rankBoth Slurm and torchrun attempted to own process creation, so rank counts, rendezvous endpoints or local device assignments no longer described one consistent process group.
First action: Print SLURM_PROCID, SLURM_LOCALID, RANK, LOCAL_RANK and CUDA_VISIBLE_DEVICES on every task, then choose one launcher to create ranks for the control run.
Full entry: root cause, fix and preventionpyxis enroot import failed to extract squashfs image: disk quota exceededEnroot exhausted the quota or inodes on its configured cache, data or temporary path while importing the container. Free capacity on another filesystem does not help that path.
First action: Print ENROOT_CACHE_PATH, ENROOT_DATA_PATH and ENROOT_TEMP_PATH inside the job, then run df -h and df -i on each resolved path before removing only unreferenced cache data.
Full entry: root cause, fix and preventionslurmstepd: error: JOB <id> CANCELLED DUE TO PREEMPTIONSlurm reclaimed the allocation for a higher-priority job according to cluster policy. The reliability failure is losing work or leaving the workflow terminal after the expected signal.
First action: Run sacct -j <jobid> --format=JobID,State,ExitCode,Elapsed and confirm the workflow treats SIGTERM or PREEMPTED as retryable only after a verified checkpoint.
Full entry: root cause, fix and preventionErrors not listed here exist. Rather than guess at their meaning, check Official Slurm GRES and job reason code references, which is the authority for the strings above.
Most of these are diagnostics rather than fixes. If one makes a failure disappear, it has told you where the fault is, not removed it.
| Variable | What it does |
|---|---|
| scontrol show job <jobid> | Shows the resolved request, state reason, time limit, node list and TRES allocation. It is the first view for rejected, pending and running jobs. |
| sacct -j <jobid> | Shows final state, exit code and measured resource use for the job and its steps. Request MaxRSS explicitly when investigating host-memory kills. |
| sinfo -R | Lists down and drained nodes with reasons. Preserve the reason before resuming a node because it often carries the only scheduler-side hardware evidence. |
| CUDA_VISIBLE_DEVICES | Slurm sets this per job step from the allocated GRES and renumbers visible devices. Application local rank must index this job-local list, not host ordinals. |
| SLURM_PROCID / SLURM_LOCALID | Identify global and node-local task indices. Print them beside RANK and LOCAL_RANK to expose duplicate launcher trees or rank mapping errors. |
| --gres=gpu:<type>:<count> | Requests generic GPU resources. The type must match the names Slurm registered, and the count participates in QOS and TRES policy. |
Which rank failed first, whether the checkpoint is safe to resume from, and whether this is your code or the hardware. Paste the log and Denpex answers all three, free and without an account.
Diagnose your logs free