A worker loses its GPU mid-run and the job hangs until a collective times out

When a GPU disappears from its host — a bus fault, a driver reset, a thermal or power event — the rank owning it stops answering. Its peers are inside a collective waiting for data that will never arrive, so the job does not crash at the moment of failure. It stalls, and the first error anyone sees is a timeout on a healthy rank several minutes later.

A stall in which one rank's accelerator becomes unreachable, leaving its peers blocked inside a collective until a timeout fires, so the failure is reported by a healthy rank long after and elsewhere from the fault.

Collectives have no liveness channel. A rank waiting for data cannot distinguish a peer that is slow from a peer that no longer exists, so it waits for its deadline and then reports what it experienced, which is a timeout. The rank that actually failed lost the device it would have needed to report with, so it says nothing. Every visible piece of evidence therefore comes from the wrong place and the wrong time.

• The reported error names a rank that is working correctly, not the one that failed
• Minutes pass between the real fault and any log line, so the timestamps mislead
• The job looks alive to the scheduler throughout the stall and keeps its allocation
• Restarting sometimes works, which suggests a transient software fault rather than hardware

• Pipeline-parallel and tensor-parallel jobs, where one lost rank stalls every rank downstream of it
• Inference engines that fan a step out to workers and wait on the result
• NVIDIA GPUs raising Xid 79, and Intel accelerators raising a level-zero device-lost result
• Any collective-based workload, since the waiting side is what reports the failure

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Compare the timestamp of the last progress line against the timestamp of the error. A gap matching the configured timeout means the job was stalled, not working, for that interval.
• ✓Search each node's kernel log for an accelerator fault at the time progress stopped; that entry names the failing device directly.
• ✓Determine which rank fell silent first from the per-rank output. That rank, not the one that raised the timeout, is where the fault is.

RuntimeError: Worker failed with error 'level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)'
NVRM: Xid (PCI:0000:41:00): 79, pid=..., GPU has fallen off the bus.
[rank2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=8412, OpType=ALLREDUCE) ran for 1800000 milliseconds before timing out

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Anchoring the investigation to the last progress line rather than the error moves it back to when the fault happened, and the host kernel log at that moment records the device event directly. That converts an unattributable job-level timeout into a named piece of hardware, which is the only form of the problem that can be acted on.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.