Skip to content

Operator task guide

Sovereign AI Slurm GH200 and H100 cluster triage

National and regional supercomputers including UK Isambard-AI and Dawn, Germany JUPITER, Switzerland CSCS Alps, Finland LUMI, France Jean Zay, Spain MareNostrum 5, Japan ABCI 3.0, Taiwan Taiwania, and Canada SciNet/Mila share parallel filesystems and tightly coupled interconnects.

Before resubmitting a large Slurm allocation, separate host cgroup OOM kills (sacct/dmesg), filesystem checkpoint hangs, single-rail InfiniBand/Slingshot link drops, and GH200/H100 Xid events so healthy nodes are not wasted on blind retries.

Reviewed . Reference guidance is not a diagnosis of your workload.

Step-by-step method

  1. 1

    Inspect Slurm step exit state and MaxRSS before inspecting PyTorch tracebacks

    Determine whether the scheduler cancelled the step on host memory (OOM_KILL), walltime, or prolog/epilog node health check failure.

    sacct -j <jobid> --format=JobID,JobName%20,NodeList,State,ExitCode,DerivedExitCode,MaxRSS,Elapsed
    scontrol show job <jobid> | rg -i 'NodeList|Reason|ExitCode'
  2. 2

    Capture NVRM Xid and NVLink/InfiniBand counters per allocated node

    Across multi-rack GH200 (NVLink-C2C) and H100 (NVSwitch + InfiniBand NDR / Slingshot) jobs, preserve exact GPU UUIDs and HCA counters before epilog node drains clear transient kernel state.

    srun --ntasks-per-node=1 --nodes=$SLURM_JOB_NUM_NODES bash -c 'hostname; dmesg -T | rg -i "nvrm|xid|mlx5|oom"; nvidia-smi -q | rg -i "UUID|Row Remap|Uncorrectable" || true'
  3. 3

    Separate shared Lustre/GPFS checkpoint stalls from fabric hangs

    If every rank blocks right at save() before emitting WorkNCCL collective timeouts, compare per-rank checkpoint flush timestamps and storage stripe contention.

    export TORCH_NCCL_TRACE_BUFFER_SIZE=2000
    export TORCH_NCCL_DUMP_ON_TIMEOUT=1
    export NCCL_DEBUG=INFO

Supercomputer Slurm hang and abort attribution matrix

Signals, meanings and actions for sovereign ai slurm gh200 and h100 cluster triage.
SignalWhat it meansNext action
sacct shows OUT_OF_MEMORY or cgroup oom_kill on one nodeHost RAM exhausted on Grace/EPYC host side or DataLoader workersLower num_workers/pin_memory host footprint; do not cordon GPU hardware.
ibv_poll_cq retry counter exceeded on single node/railInfiniBand optical link, HCA port, or subnet congestion faultRecord hostname and mlx5 port index and request node drain from cluster ops.
Xid 79, Xid 74, or Xid 149 on one GPU UUIDDevice PCIe/NVLink loss or microcontroller halt prior to collective abortOpen support ticket with UUID, PCI bus ID, and step logs rather than retrying on the same node.
Watchdog caught collective timeout across hundreds of nodesCollateral timeout while waiting for an already-failed peer or slow storage rankDiff WorkNCCL SeqNum across ranks to locate the earliest stalled rank.

Evidence checklist

  • Complete sacct and scontrol job step metadata
  • Host dmesg kernel events across all allocated nodes
  • NCCL_DEBUG=INFO and Flight Recorder timeout traces
  • GPU UUID, serial, and per-node IB / Slingshot port status

Common mistakes

Re-queuing immediately onto the same unhealthy Slurm partition

Without excluding the failing node via -x node, multi-node jobs frequently land back on the degraded HCA or GPU and burn allocation credits.

Assuming WorkNCCL Rank 0 caused the timeout

Rank 0 logs loudest because it coordinates the collective; always check which node exited silently or lagged in SeqNum.

Frequently asked questions

Why do GH200 Slurm jobs hit host OOM before GPU VRAM is full?

GH200 tightly couples Grace LPDDR5X host memory and Hopper HBM3e. Slurm cgroup limits enforce unified host+step RSS bounds, so large pinned DataLoader buffers or core dumps can trip oom_kill while GPU HBM still has headroom.

Can air-gapped sovereign clusters use Denpex without uploading logs?

Yes. The /tools/gpu-log-analyzer browser tool and local CLI run deterministically with zero external log egress.

Apply the method to your incident

Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.

Diagnose your incident