Xid 79: GPU has fallen off the bus

Xid 79 means the NVIDIA driver cannot reach the GPU over PCIe. Investigate link, device, and driver evidence before deciding on recovery or replacement.

Symptom

Xid 79: GPU has fallen off the bus

Root cause

A PCIe link failure can make the endpoint inaccessible. GPU hardware failure can produce the same event. Driver issues can also cause Xid 79; the event is not proof of an exclusively electrical fault.

Recommended fix

Preserve kernel logs, the event timestamp, GPU identity, and an NVIDIA bug report before disruptive recovery where possible.

How Denpex helps

Denpex matches Xid 79: GPU has fallen off the bus across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

NVIDIA emits Xid 79 when an attempted PCIe access finds the GPU inaccessible. It identifies a loss of device accessibility, not a complete diagnosis of the underlying component.

NVIDIA identifies PCIe link failures, failing GPU hardware, and other driver issues as possible explanations. Correlate the event with kernel PCIe logs, platform events, driver version, and the affected device identity. Power and thermal records can help investigate a platform fault but do not establish one merely because Xid 79 occurred.

• A job loses access to a GPU
• The driver reports that the GPU is no longer accessible over PCIe
• The fault may recur after recovery and require vendor investigation

• NVIDIA GPUs and supported Linux drivers
• Bare-metal GPU hosts and virtualized environments where host evidence is available

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Locate Xid 79 in the host kernel log
• ✓Record the affected PCI bus address and GPU identity
• ✓Compare PCIe and system event logs around the incident
• ✓Record recovery results and whether the problem recurs

• A PCIe link failure can make the endpoint inaccessible
• GPU hardware failure can produce the same event
• Driver issues can also cause Xid 79; the event is not proof of an exclusively electrical fault

Xid 79: GPU has fallen off the bus

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Evidence collection preserves the distinction between a device-access failure and its cause. A coordinated restart addresses the immediate host condition while subsequent diagnostics and recurrence evidence guide the vendor investigation.

# Read-only evidence collection. Run with the permissions required by your host.
dmesg -T | grep -iE 'NVRM|Xid|pcieport|AER'
nvidia-smi -L
lspci -nn
# Collect the vendor report before recovery where possible.
sudo nvidia-bug-report.sh
# Review the report for sensitive host details before sharing it.

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.

Xid 79 is one of a small set of codes the NVIDIA driver uses to report GPU faults, and the number is most of the diagnosis: it tells you whether you are looking at your own code, the driver, or a board that needs replacing.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.