Torchrun manages the lifecycle of distributed PyTorch processes, handling rank assignment and initial rendezvous. When torchrun fails, it usually means the nodes could not discover each other on the network.
The most critical debugging step is isolating whether the failure is a network connectivity issue between the master and worker nodes, or a configuration mismatch in the expected world size. Always check the rendezvous endpoint first.
Working in VS Code or Remote-SSH? Diagnose your log locally with the free VS Code extension. No account or upload required for local diagnosis.
The literal string is what you paste into a search bar, so it is the heading. “Class” is a triage category. It is not proof of ownership in your workload.
RuntimeError: The server socket has failed to listen on any local network addressThe master node could not bind to the specified port for the C10d rendezvous backend. The port is either already in use by another process or the network interface is down.
First action: Run python -m torch.distributed.run to verify local syntax, check for port conflicts, and ensure the --master_addr network interface is up.
ConnectionRefusedError: [Errno 111] Connection refused in rendezvousA worker node attempted to connect to the master node for rendezvous, but the connection was actively rejected. The master may have crashed, or a firewall is blocking the port.
First action: Verify the master process is actually running before debugging network rules. Test connectivity to the --master_addr IP and port from the worker node.
TimeoutError: Rendezvous timeout after 900 secondsNot all expected nodes joined the rendezvous group before the timeout expired. The world size was configured for N nodes, but fewer than N nodes successfully connected.
First action: Count the number of active worker nodes and compare it to the --nnodes argument in your torchrun command. A node that failed to boot causes this timeout.
Errors not listed here exist. Rather than guess at their meaning, check PyTorch Elastic/Torchrun Docs, which is the authority for the strings above.
Which rank failed first, whether the checkpoint is safe to resume from, and whether this is your code or the hardware. Paste the log and Denpex answers all three, free and without an account.
Diagnose your logs free