Slurm vs Kubernetes GPU Training Failures & Triage
Orchestrating GPU training jobs requires robust workload managers. Slurm and Kubernetes handle node allocation, device plugin mapping, and failure recovery through completely different mechanisms. This guide contrasts their failure signatures and triage workflows.
Comparison Matrix
| Dimension | Slurm Workload Manager | Kubernetes (K8s + GPU Operator) | Shared Training Challenges |
|---|---|---|---|
| Node & GPU Allocation | GRES (Generic Resources) allocation via srun / sbatch script headers. | NVIDIA GPU Operator, Device Plugin, and Kueue / Volcano queue scheduling. | Distributed rendezvous timeout and rank coordination across nodes. |
| Job Failure Signatures | slurmstepd: JOB 4192 CANCELLED DUE TO OOM-KILLER or node drain event. | Pod evicted, OOMKilled (Exit Code 137), or ImagePullBackOff on worker pods. | Rank connection refused, c10d socket timeout, or half-dead node freezing job. |
| Process & Tree Management | Direct process supervision via slurmstepd and cgroup v2 task containment. | Container runtime isolation (containerd) with cgroup limits per pod container. | Zombie process accumulation and orphaned torchrun subprocesses. |
| Node Draining & Recovery | Automatic node draining via Epilog Slurm scripts upon Xid error detection. | Kubernetes node drain, taint/toleration automation, and cluster autoscaler hooks. | Isolating faulty GPUs before rescheduling distributed checkpoint restarts. |
| Debugging CLI Tooling | squeue, sinfo, seff, sacct, and journalctl -u slurmd. | kubectl get pods, kubectl describe pod, kubectl logs, and crictl. | GPU log analyzers and multi-rank timeline correlation engines. |
Signature failures and first triage commands
Slurm Job Cancelled Due to OOM
slurmstepd: error: job 4192 exceeded memory limit (1024000 > 1000000), expelled and cancelled.seff 4192 && sacct -j 4192 --format=JobID,JobName,State,ExitCode,MaxRSSKubernetes Pod OOMKilled (Exit Code 137)
Normal Scheduled ... Warning Unhealthy ... Last State: Terminated | Reason: OOMKilled | Exit Code: 137kubectl describe pod training-worker-0GPU Operator Device Plugin Failure
E0412 12:00:00.123456 main.go:100] Failed to allocate GPUs: error connecting to NVML library.kubectl logs -n gpu-operator daemonset/nvidia-device-plugin-daemonsetEngineering decision guide
Choose Slurm for High-Performance Computing & Raw Bare-Metal Scale
Slurm remains the gold standard for tightly-coupled MPI and large-scale GPU training clusters, offering low overhead and native HPC job step accounting.
Choose Kubernetes for Cloud-Native Flexibility & Microservice Co-location
Kubernetes enables dynamic multi-tenant scheduling, seamless integration with cloud storage and web services, and automated elastic scaling using operators like Kueue.
Standardize Failure Response Across Both
Regardless of whether you use sbatch or kubectl apply, automated Xid detection and rapid node draining are mandatory to prevent cascading GPU hardware failures.
Frequently asked questions
How does Slurm GRES differ from Kubernetes NVIDIA GPU device plugin?
Slurm GRES allocates specific physical GPU device files directly to job steps via cgroups. Kubernetes uses the NVIDIA Device Plugin to advertise nvidia.com/gpu resources to the kubelet, mounting device files into container namespaces.
What causes Exit Code 137 in Kubernetes GPU training pods?
Exit Code 137 means the container was forcefully terminated by the Linux kernel OOM killer because its memory consumption exceeded the pod request/limit specification.
How can I automatically drain a Slurm node when an Xid error occurs?
Configure a Prolog/Epilog health check script or integrate an external monitoring daemon (such as Denpex or custom DCGM watchdogs) that executes 'scontrol update NodeName=node state=DRAIN reason=Xid_Error'.