Torchrun and SLURM Job Step Incompatibility
Torchrun's approach to spawning jobs conflicts with SLURM's process management, resulting in hung NCCL initialization, rank mismapping, and job allocation failures.
Torchrun's approach to spawning jobs conflicts with SLURM's process management, resulting in hung NCCL initialization, rank mismapping, and job allocation failures.
What this failure is
Torchrun and SLURM Job Step Incompatibility is a Distributed Training failure seen during ML training runs. Torchrun's approach to spawning jobs conflicts with SLURM's process management, resulting in hung NCCL initialization, rank mismapping, and job allocation failures. Common tags: Slurm, Torchrun, Distributed, Nccl.
Is this what broke your run? Paste your log.
You're reading about Torchrun and SLURM Job Step Incompatibility. Paste your own crash log or traceback below and get the real root cause for YOUR run, not this generic entry. No account, no card. Logs are masked at ingress and never saved to account history.
Want 14 days on the Scale plan?
Request an evaluation code. A verified workplace organization activates up to 50 diagnoses a day, alerts, history, and follow-up questions. No credit card or automatic subscription.
Why it happens (the mechanism)
SLURM manages job steps independently, while Torchrun attempts to spawn its own child processes, causing environment variables like SLURM_PROCID to desync. Multiple torchrun instances on the same node bind to the same rendezvous port. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.
What you'll observe
- NCCL hangs indefinitely during initialization without throwing an error
- Ranks are mapped incorrectly across nodes, causing tensor parallel deadlocks
- SLURM jobs crash with 'Connection refused' during Torch distributed rendezvous
Common symptoms and what they mean
| Symptom | Why it happens |
|---|---|
| Torchrun elastic agent fails to discover all peers | SLURM manages job steps independently, while Torchrun attempts to spawn its own child processes, causing environment variables like SLURM_PROCID to desync |
| SLURM_PROCID does not match the expected LOCAL_RANK inside the container | Multiple torchrun instances on the same node bind to the same rendezvous port |
| Process exits with code 137 or timeout during init_process_group | SLURM manages job steps independently, while Torchrun attempts to spawn its own child processes, causing environment variables like SLURM_PROCID to desync |
Which systems are affected
- HPC clusters using SLURM workload manager
- Multi-node PyTorch training jobs launched via srun + torchrun
How to confirm this is the problem
Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.
- ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
- ✓Verified signal present: Torchrun elastic agent fails to discover all peers
- ✓Verified signal present: SLURM_PROCID does not match the expected LOCAL_RANK inside the container
- ✓Verified signal present: Process exits with code 137 or timeout during init_process_group
- ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.
The fix and the prevention pattern
The root cause is on this page and stays free. A free account adds the exact remediation steps, saved history, and the fix on every entry in the encyclopedia.
Sign up free. Unlock the full analysisNo credit card. Daily allowance follows verified trust tier. Instant access.
Diagnose this failure in VS Code
Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.
Install the free VS Code extensionSlurm GPU errors in context
Slurm GPU failures can occur before allocation, during cgroup creation or inside the workload. The hub joins job reason, GRES, memory, node-health and distributed-launch evidence.
Compare every slurm gpu error side by sideRelated failures to investigate next
Root cause
- SLURM manages job steps independently, while Torchrun attempts to spawn its own child processes, causing environment variables like SLURM_PROCID to desync
- Multiple torchrun instances on the same node bind to the same rendezvous port
The fix and how to prevent it
Evaluate Denpex on your own logs
Request a Scale evaluation code. A verified workplace organization activates 14 days with up to 50 diagnoses a day. Every account keeps its current diagnosis allowance and gets a verification path. No card or automatic subscription.
Don't just read the fix, diagnose your run
The encyclopedia tells you what went wrong. Denpex tells you what went wrong in YOUR training run. With your logs, your config, and your stack.