Re-queuing immediately onto the same unhealthy Slurm partition
Without excluding the failing node via -x node, multi-node jobs frequently land back on the degraded HCA or GPU and burn allocation credits.
Operator task guide
National and regional supercomputers including UK Isambard-AI and Dawn, Germany JUPITER, Switzerland CSCS Alps, Finland LUMI, France Jean Zay, Spain MareNostrum 5, Japan ABCI 3.0, Taiwan Taiwania, and Canada SciNet/Mila share parallel filesystems and tightly coupled interconnects.
Before resubmitting a large Slurm allocation, separate host cgroup OOM kills (sacct/dmesg), filesystem checkpoint hangs, single-rail InfiniBand/Slingshot link drops, and GH200/H100 Xid events so healthy nodes are not wasted on blind retries.
Reviewed . Reference guidance is not a diagnosis of your workload.
Determine whether the scheduler cancelled the step on host memory (OOM_KILL), walltime, or prolog/epilog node health check failure.
sacct -j <jobid> --format=JobID,JobName%20,NodeList,State,ExitCode,DerivedExitCode,MaxRSS,Elapsed
scontrol show job <jobid> | rg -i 'NodeList|Reason|ExitCode'Across multi-rack GH200 (NVLink-C2C) and H100 (NVSwitch + InfiniBand NDR / Slingshot) jobs, preserve exact GPU UUIDs and HCA counters before epilog node drains clear transient kernel state.
srun --ntasks-per-node=1 --nodes=$SLURM_JOB_NUM_NODES bash -c 'hostname; dmesg -T | rg -i "nvrm|xid|mlx5|oom"; nvidia-smi -q | rg -i "UUID|Row Remap|Uncorrectable" || true'If every rank blocks right at save() before emitting WorkNCCL collective timeouts, compare per-rank checkpoint flush timestamps and storage stripe contention.
export TORCH_NCCL_TRACE_BUFFER_SIZE=2000
export TORCH_NCCL_DUMP_ON_TIMEOUT=1
export NCCL_DEBUG=INFO| Signal | What it means | Next action |
|---|---|---|
| sacct shows OUT_OF_MEMORY or cgroup oom_kill on one node | Host RAM exhausted on Grace/EPYC host side or DataLoader workers | Lower num_workers/pin_memory host footprint; do not cordon GPU hardware. |
| ibv_poll_cq retry counter exceeded on single node/rail | InfiniBand optical link, HCA port, or subnet congestion fault | Record hostname and mlx5 port index and request node drain from cluster ops. |
| Xid 79, Xid 74, or Xid 149 on one GPU UUID | Device PCIe/NVLink loss or microcontroller halt prior to collective abort | Open support ticket with UUID, PCI bus ID, and step logs rather than retrying on the same node. |
| Watchdog caught collective timeout across hundreds of nodes | Collateral timeout while waiting for an already-failed peer or slow storage rank | Diff WorkNCCL SeqNum across ranks to locate the earliest stalled rank. |
Without excluding the failing node via -x node, multi-node jobs frequently land back on the degraded HCA or GPU and burn allocation credits.
Rank 0 logs loudest because it coordinates the collective; always check which node exited silently or lagged in SeqNum.
Reference common Slurm exit codes, cgroup kills, and node drain signals.
Analyze multi-rank Slurm and NCCL logs locally with zero egress for air-gapped environments.
Package clean hardware versus node evidence for supercomputer support desks.
GH200 tightly couples Grace LPDDR5X host memory and Hopper HBM3e. Slurm cgroup limits enforce unified host+step RSS bounds, so large pinned DataLoader buffers or core dumps can trip oom_kill while GPU HBM still has headroom.
Yes. The /tools/gpu-log-analyzer browser tool and local CLI run deterministically with zero external log egress.
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident