A worker loses its GPU mid-run and the job hangs until a collective times out

When a GPU disappears from its host, a bus fault, a driver reset, a thermal or power event, the rank owning it stops answering. Its peers are inside a collective waiting for data that will never arrive, so the job does not crash at the moment of failure. It stalls, and the first error anyone sees is a timeout on a healthy rank several minutes later.

Symptom

RuntimeError: Worker failed with error 'level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)'

Root cause

A collective is a rendezvous. Every participant blocks until all of them arrive, and there is no mechanism by which a waiting rank learns that a peer has ceased to exist, it can only observe that the data has not come. The absence of a peer and a peer that is merely slow are indistinguishable until a deadline expires.

Recommended fix

Take the timestamp of the last normal progress line, not the timestamp of the error, and look in the host kernel log of every node at that moment. The device-loss event is recorded there when it happens.

How Denpex helps

Denpex matches A worker loses its GPU mid-run and the job hangs until a collective times out across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

A stall in which one rank's accelerator becomes unreachable, leaving its peers blocked inside a collective until a timeout fires, so the failure is reported by a healthy rank long after and elsewhere from the fault.

Collectives have no liveness channel. A rank waiting for data cannot distinguish a peer that is slow from a peer that no longer exists, so it waits for its deadline and then reports what it experienced, which is a timeout. The rank that actually failed lost the device it would have needed to report with, so it says nothing. Every visible piece of evidence therefore comes from the wrong place and the wrong time.

• The reported error names a rank that is working correctly, not the one that failed
• Minutes pass between the real fault and any log line, so the timestamps mislead
• The job looks alive to the scheduler throughout the stall and keeps its allocation
• Restarting sometimes works, which suggests a transient software fault rather than hardware

• Pipeline-parallel and tensor-parallel jobs, where one lost rank stalls every rank downstream of it
• Inference engines that fan a step out to workers and wait on the result
• NVIDIA GPUs raising Xid 79, and Intel accelerators raising a level-zero device-lost result
• Any collective-based workload, since the waiting side is what reports the failure

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Compare the timestamp of the last progress line against the timestamp of the error. A gap matching the configured timeout means the job was stalled, not working, for that interval.
• ✓Search each node's kernel log for an accelerator fault at the time progress stopped; that entry names the failing device directly.
• ✓Determine which rank fell silent first from the per-rank output. That rank, not the one that raised the timeout, is where the fault is.

• A collective is a rendezvous. Every participant blocks until all of them arrive, and there is no mechanism by which a waiting rank learns that a peer has ceased to exist, it can only observe that the data has not come. The absence of a peer and a peer that is merely slow are indistinguishable until a deadline expires.
• So the failure inverts the usual relationship between cause and report. The rank whose device vanished cannot report anything, because the thing it would report with is gone. The report comes from a healthy rank, describing a symptom of someone else's fault, at a time chosen by the timeout rather than by the event.
• The device loss itself is a host-level event. It is recorded in the kernel log at the instant it happens, which is why that log and the job's own output disagree about when the failure occurred, often by the whole length of the timeout.

RuntimeError: Worker failed with error 'level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)'
NVRM: Xid (PCI:0000:41:00): 79, pid=..., GPU has fallen off the bus.
[rank2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=8412, OpType=ALLREDUCE) ran for 1800000 milliseconds before timing out

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Anchoring the investigation to the last progress line rather than the error moves it back to when the fault happened, and the host kernel log at that moment records the device event directly. That converts an unattributable job-level timeout into a named piece of hardware, which is the only form of the problem that can be acted on.

Xid 79 is one of a small set of codes the NVIDIA driver uses to report GPU faults, and the number is most of the diagnosis: it tells you whether you are looking at your own code, the driver, or a board that needs replacing.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.