Automatic Node Failure Detection and Training Resumption Using Node Doctor and Watchdog

MosaicML's training platform automatically detected hardware failures and resumed training without human intervention using its Node Doctor and Watchdog services. During Stable Diffusion training on 128 A100 GPUs, these systems eliminated the need for 24/7 human babysitting by automatically capturing failure signals, draining failed nodes, and resuming from the last checkpoint. This pattern was critical for achieving production reliability at sub-$50k training costs.

Automatic Node Failure Detection and Training Resumption Using Node Doctor and Watchdog is a Reliability failure seen during ML training runs. MosaicML's training platform automatically detected hardware failures and resumed training without human intervention using its Node Doctor and Watchdog services. During Stable Diffusion training on 128 A100 GPUs, these systems eliminated the need for 24/7 human babysitting by automatically capturing failure signals, draining failed nodes, and resuming from the last checkpoint. This pattern was critical for achieving production reliability at sub-$50k training costs. Common tags: Mosaicml, Databricks, Auto Resumption, Node Doctor.

GPU hardware fault, network partition, or spot instance preemption causes one node to become unavailable during training. Synchronous distributed training cannot progress past the failing rank's last all-reduce barrier. Human-in-the-loop recovery introduces 5-30 minute gap between failure detection and resumption. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Hardware failures during multi-week training require 24/7 human monitoring or waste GPU hours in idle restart
• Single node failure in a 128-GPU job stalls all GPUs until manual intervention detects and resolves
• Checkpoint-then-restart workflow requires manual verification that resumption succeeded

• Multi-node GPU training on spot or preemptible instances
• Long-running training without dedicated ops team monitoring
• PyTorch FSDP / DDP training with standard checkpoint-restart recovery

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: One of 128 GPUs stops producing output while remaining GPUs block on NCCL collective
• ✓Verified signal present: DCGM shows GPU health check failure or XID on a single node
• ✓Verified signal present: Training progress stalls for minutes to hours while operator is paged and responds
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• GPU hardware fault, network partition, or spot instance preemption causes one node to become unavailable during training
• Synchronous distributed training cannot progress past the failing rank's last all-reduce barrier
• Human-in-the-loop recovery introduces 5-30 minute gap between failure detection and resumption