NVIDIA Xid 79: GPU Has Fallen Off the Bus Fix
GPU has fallen off the bus (NVIDIA Xid 79): preserve kernel logs and check PCIe, power and host evidence before deciding on recovery or hardware replacement.
Xid 79 means the NVIDIA driver lost access to a GPU over PCIe. Before restarting, save the full host kernel event and timestamp, host or instance ID, PCI bus address and GPU UUID. Check nearby PCIe/AER, power and platform events. On rented infrastructure, send this packet to the provider so they can inspect the host and decide the recovery action.
- Root cause
- A PCIe link failure can make the endpoint inaccessible. GPU hardware failure can produce the same event. Driver issues can also cause Xid 79; the event is not proof of an exclusively electrical fault.
- Recommended fix
- Preserve kernel logs, the event timestamp, GPU identity, and an NVIDIA bug report before disruptive recovery where possible.
- How Denpex helps
- Denpex investigates NVIDIA Xid 79: GPU Has Fallen Off the Bus Fix using the evidence you provide or your connected workload collects. Earlier rank, host or application evidence is needed to distinguish an initiating failure from a downstream report.
What this failure is
NVIDIA emits Xid 79 when an attempted PCIe access finds the GPU inaccessible. It identifies a loss of device accessibility, not a complete diagnosis of the underlying component.
Is this what broke your run? Paste your log.
You're reading about NVIDIA Xid 79: GPU Has Fallen Off the Bus Fix. Paste your own traceback and relevant evidence for an investigation of your workload, with a next action or a specific missing fact. A reference entry does not establish your cause. No account or card for the free diagnosis. Review data handling before submitting sensitive logs.
Before uploading, review cloud data handling and local options.
Upload your log file. Large files are processed in your browser. Only selected evidence is submitted when you run diagnosis.
Why it happens (the mechanism)
NVIDIA identifies PCIe link failures, failing GPU hardware, and other driver issues as possible explanations. Correlate the event with kernel PCIe logs, platform events, driver version, and the affected device identity. Power and thermal records can help investigate a platform fault but do not establish one merely because Xid 79 occurred.
What you'll observe
- A job loses access to a GPU
- The driver reports that the GPU is no longer accessible over PCIe
- The fault may recur after recovery and require vendor investigation
Common symptoms and what they mean
Check possible explanations against the incident evidence. A matching symptom alone does not establish its cause.
| Symptom | Possible explanation |
|---|---|
| NVRM: Xid with code 79 and the message GPU has fallen off the bus | A PCIe link failure can make the endpoint inaccessible |
| GPU enumeration or management commands may fail for the affected device | GPU hardware failure can produce the same event |
| Kernel PCIe events may provide additional context | Driver issues can also cause Xid 79; the event is not proof of an exclusively electrical fault |
Which systems are affected
- NVIDIA GPUs and supported Linux drivers
- Bare-metal GPU hosts and virtualized environments where host evidence is available
How to confirm this is the problem
Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.
- ✓Find the complete Xid 79 event in the host kernel log and retain nearby PCIe/AER and platform events.
- ✓Map the PCI bus address and GPU UUID to the affected host and job. A container GPU index can differ from the host index.
- ✓Separate a host that cannot access its GPU from a container that cannot access an otherwise working host device. NVML initialization errors alone do not establish Xid 79.
- ✓For a rented GPU, send the provider the event time with timezone, instance/node identity, original error and any permitted device identifiers. Ask the operator for host-side evidence before requesting disruptive recovery.
- ✓Record the operator recovery action, subsequent device/fabric state, workload results and recurrence. Absence of AER messages alone does not certify a healthy link.
Root cause
- A PCIe link failure can make the endpoint inaccessible
- GPU hardware failure can produce the same event
- Driver issues can also cause Xid 79; the event is not proof of an exclusively electrical fault
The fix and how to prevent it
How to fix it
- Preserve kernel logs, the event timestamp, GPU identity, and an NVIDIA bug report before disruptive recovery where possible.
- Drain affected workloads and coordinate the restart with the host owner. NVIDIA's catalog lists a bare-metal host restart as the immediate action for Xid 79.
- Use the OEM or cloud provider recovery procedure. A full power cycle may be required on some platforms, but do not claim that every warm reboot must fail.
- Review PCIe events, platform logs, and the supported driver and firmware baseline. Escalate recurring or unresolved faults with the evidence to NVIDIA, the OEM, or the cloud provider.
- Run approved diagnostics after draining work and verify that the GPU is usable before returning the host to service. Do not replace hardware based only on the Xid number.
Diagnose your own log automatically
The root cause and fix steps above stay free. Create a free account to analyze full job logs, attribute the first failed rank, and keep saved incident history.
Sign up free. Analyze your cluster logsNo credit card. Daily allowance follows verified trust tier. Instant access.
Why the recommended fix works
Evidence collection preserves the distinction between a device-access failure and its cause. A coordinated restart addresses the immediate host condition while subsequent diagnostics and recurrence evidence guide the vendor investigation.
Diagnostic note
“Illustrative timeline only: a host logs Xid 79 at 02:14:08 UTC and a peer reports an NCCL timeout at 02:15:08 UTC. The event order supports investigating lost device access before the peer timeout. It does not identify whether the card, link, platform or software caused it. Do not treat this illustrative timeline as a measured customer recovery.”
Xid 79 in context
Xid 79 identifies a driver-observed event. Preserve the full message, device identity and surrounding workload and host evidence before distinguishing application, driver and hardware causes. The code alone does not justify replacing a board.
Compare Xid event familiesDiagnose this failure in VS Code
Select the traceback or open the failed terminal, then run Denpex locally to inspect matched failure patterns, supporting evidence and next checks without uploading the log. Verify recommendations against your workload before changing production.
Install the free VS Code extensionRelated failures to investigate next
Frequently asked questions
Questions engineers and on-call staff commonly ask about this failure.
Does Xid 79 prove the card needs replacement?
How do I diagnose this on a rented GPU?
What if nvidia-smi works on the host but fails in a container?
Don't just read the fix, diagnose your run
The encyclopedia explains documented failure patterns. Denpex investigates your logs, configuration and stack, shows the supporting evidence, and asks for missing details when needed.