Operator task guide

How to diagnose a Slurm GPU OOM

First identify which memory boundary failed. A slurmstepd OOM-killer message usually means the job step exceeded host RAM enforced by its cgroup. A framework CUDA out-of-memory exception means GPU framebuffer allocation failed. They require different evidence and different fixes.

Use sacct and scontrol to recover the effective request and MaxRSS, inspect the host cgroup and kernel for killed processes, then compare nvidia-smi and the first rank-local exception. Do not increase both host memory and GPU batch settings until the failed boundary is proven.

Step-by-step method

1
Capture the exact first OOM signature
Preserve the first message and its emitting process. Later NCCL timeouts and launcher errors are expected after one rank is killed and should not replace the OOM as the initiating evidence.
```
rg -n -i 'oom|out of memory|killed process|cuda out of memory|nccl|childfailederror' slurm-*.out
```
2
Read Slurm accounting and the resolved request
Compare requested memory with observed MaxRSS for the job and each step. Record whether the job used --mem, --mem-per-cpu or --mem-per-gpu and whether the partition or QOS changed the effective boundary.
```
sacct -j <jobid> --units=G --format=JobID,JobName,State,ExitCode,ReqMem,MaxRSS,MaxVMSize,NodeList
scontrol show job <jobid>
```
3
Confirm a host cgroup kill
A slurmstepd cancellation plus a kernel or memory.events kill confirms host-memory enforcement. Offload, data loaders, page cache, pinned buffers and per-rank duplication all consume host RAM even when HBM is available.
```
dmesg -T | rg -i 'oom|killed process|memory cgroup'
cat /sys/fs/cgroup/<job-step>/memory.events
```
4
Confirm a CUDA device-memory failure
A PyTorch or CUDA OOM is a device allocation failure. Compare allocated, reserved and free HBM on the failing rank, then inspect batch, sequence, activation, model-state and fragmentation conditions.
```
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv
nvidia-smi --query-gpu=index,uuid,memory.used,memory.total --format=csv
```
5
Separate step creation from runtime OOM
If slurmstepd says it could not allocate the requested memory cgroup before the process started, compare the request with node RealMemory and cgroup configuration. There is no application memory profile yet because the workload never ran.
```
scontrol show node <node> | rg 'RealMemory|CfgTRES|AllocTRES|State'
scontrol show config | rg -i 'selecttypeparameters|taskplugin|proctracktype'
```
6
Repeat one boundary-specific control
For host OOM, change loader workers, offload, pinning or requested host memory. For CUDA OOM, change device batch, sequence, checkpointing or allocator behavior. Hold the other boundary constant and require the initiating signature to disappear.

Slurm GPU OOM decision table

Signals, meanings and actions for how to diagnose a slurm gpu oom.
Signal	What it means	Next action
slurmstepd: JOB CANCELLED DUE TO OOM-KILLER	Host memory cgroup was exceeded	Use sacct MaxRSS, memory.events and kernel evidence. Review host request and per-rank duplication.
torch.OutOfMemoryError: CUDA out of memory	GPU framebuffer allocation failed	Inspect HBM allocation, batch, sequence, activations, model state and fragmentation on the first rank.
task/cgroup: unable to allocate requested memory	Step memory boundary could not be created	Compare request with node RealMemory and the cgroup configuration before debugging the application.
NCCL watchdog timeout after one OOM	Other ranks waited after the failed rank disappeared	Treat the first OOM as initiating evidence and the timeout ranks as the cascade.
Container /dev/shm is 64 MB	Shared-memory limit can break local collectives	Measure and correct /dev/shm separately. It is not fixed by adding HBM or Slurm --mem alone.

Evidence checklist

Exact first OOM line with rank, node, process and surrounding context
sacct job and step states, exit codes, ReqMem and MaxRSS
scontrol resolved job request and node RealMemory
Host kernel OOM and cgroup memory.events evidence
GPU UUID, process memory and framework allocator output
One boundary-specific control with all other variables held constant

Common mistakes

Calling every Slurm OOM a CUDA OOM

Slurm cgroups enforce host RAM. GPU memory is a separate allocator and usually reports through the framework.

Looking only at the batch step

Launcher, extern and child steps can carry different states and MaxRSS. Inspect the full sacct tree.

Treating NCCL timeout as a second root cause

Once one rank is killed by an OOM, surviving ranks predictably time out in collectives. Fix the initiating memory boundary first.

Tools and deeper references

Slurm GPU error reference

Decode cgroup, QOS, pending, drained-node and time-limit failures.

Slurm OOM-killer entry

Follow the exact job-cancelled signature and verification steps.

CUDA error reference

Diagnose device OOM and distinguish capacity from fragmentation.

HPC and Slurm reliability

Connect job support, local diagnosis, scheduler evidence and outcome metrics.

Official references

Frequently asked questions

Does Slurm OOM mean the GPU ran out of memory?

Usually not when the message comes from slurmstepd or the OOM killer. That path enforces host memory through the job cgroup. A CUDA device-memory OOM is emitted by the framework or CUDA allocator and should include device allocation details.

Why do other ranks report NCCL timeouts after one Slurm OOM?

The killed rank can no longer enter the collective. Surviving ranks remain alive, wait for it and eventually trigger watchdog timeouts. The first host OOM evidence owns the incident unless an even earlier application failure caused the memory growth.

Should I increase --mem or --mem-per-gpu?

Only after confirming host-memory pressure and checking the site's accounting policy. More requested memory can be appropriate, but data-loader workers, offload duplication, pinned buffers or a leak may still make use scale unexpectedly. Measure MaxRSS and the per-rank model first.

Can CUDA memory be free while Slurm kills the job?

Yes. HBM and host RAM are separate resources. A job can have free device memory while CPU offload, data loading, page cache or duplicated process state exceeds the Slurm memory cgroup and triggers a host OOM kill.

Apply the method to the complete incident

Paste the logs for an answer-first diagnosis, evidence request and verification step.

Diagnose the incident