Network Partition

Network partitions split distributed training into isolated groups, causing hangs, deadlocks, or wrong gradients.

Network Partition is a Reliability failure seen during ML training runs. Network partitions split distributed training into isolated groups, causing hangs, deadlocks, or wrong gradients. Common tags: Network, Partition, Hang, Distributed.

Network interface goes down. Switch/router failure. Network congestion causes timeouts. Subnet issues between nodes. Firewall rules changing mid-training. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Distributed training hangs without error
• Some workers don't see all-reduce
• NCCL operations time out

• Multi-node training across data centers
• Training on spot instances with network instability
• Training with imperfect network connectivity

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: NCCL WARN unhandled cuda error
• ✓Verified signal present: NCCL timeout waiting for operation
• ✓Verified signal present: torch.distributed barrier hangs
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Network interface goes down
• Switch/router failure
• Network congestion causes timeouts
• Subnet issues between nodes
• Firewall rules changing mid-training