Skip to content

NVIDIA Xid 79: GPU Has Fallen Off the Bus Fix

GPU has fallen off the bus (NVIDIA Xid 79): preserve kernel logs and check PCIe, power and host evidence before deciding on recovery or hardware replacement.

Quick answer

Xid 79 means the NVIDIA driver lost access to a GPU over PCIe. Before restarting, save the full host kernel event and timestamp, host or instance ID, PCI bus address and GPU UUID. Check nearby PCIe/AER, power and platform events. On rented infrastructure, send this packet to the provider so they can inspect the host and decide the recovery action.

Root cause
A PCIe link failure can make the endpoint inaccessible. GPU hardware failure can produce the same event. Driver issues can also cause Xid 79; the event is not proof of an exclusively electrical fault.
Recommended fix
Preserve kernel logs, the event timestamp, GPU identity, and an NVIDIA bug report before disruptive recovery where possible.
How Denpex helps
Denpex investigates NVIDIA Xid 79: GPU Has Fallen Off the Bus Fix using the evidence you provide or your connected workload collects. Earlier rank, host or application evidence is needed to distinguish an initiating failure from a downstream report.
Hardware#xid#xid-79#pcie#nvidia-driver#hardware#rma

What this failure is

NVIDIA emits Xid 79 when an attempted PCIe access finds the GPU inaccessible. It identifies a loss of device accessibility, not a complete diagnosis of the underlying component.

Live diagnosis, no signup

Is this what broke your run? Paste your log.

You're reading about NVIDIA Xid 79: GPU Has Fallen Off the Bus Fix. Paste your own traceback and relevant evidence for an investigation of your workload, with a next action or a specific missing fact. A reference entry does not establish your cause. No account or card for the free diagnosis. Review data handling before submitting sensitive logs.

Before uploading, review cloud data handling and local options.

training_logs.txt
No log to hand? Try one:

Upload your log file. Large files are processed in your browser. Only selected evidence is submitted when you run diagnosis.

3 free cloud diagnoses per day

Why it happens (the mechanism)

NVIDIA identifies PCIe link failures, failing GPU hardware, and other driver issues as possible explanations. Correlate the event with kernel PCIe logs, platform events, driver version, and the affected device identity. Power and thermal records can help investigate a platform fault but do not establish one merely because Xid 79 occurred.

What you'll observe

  • A job loses access to a GPU
  • The driver reports that the GPU is no longer accessible over PCIe
  • The fault may recur after recovery and require vendor investigation

Common symptoms and what they mean

Check possible explanations against the incident evidence. A matching symptom alone does not establish its cause.

SymptomPossible explanation
NVRM: Xid with code 79 and the message GPU has fallen off the busA PCIe link failure can make the endpoint inaccessible
GPU enumeration or management commands may fail for the affected deviceGPU hardware failure can produce the same event
Kernel PCIe events may provide additional contextDriver issues can also cause Xid 79; the event is not proof of an exclusively electrical fault

Which systems are affected

  • NVIDIA GPUs and supported Linux drivers
  • Bare-metal GPU hosts and virtualized environments where host evidence is available

How to confirm this is the problem

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

  • ✓Find the complete Xid 79 event in the host kernel log and retain nearby PCIe/AER and platform events.
  • ✓Map the PCI bus address and GPU UUID to the affected host and job. A container GPU index can differ from the host index.
  • ✓Separate a host that cannot access its GPU from a container that cannot access an otherwise working host device. NVML initialization errors alone do not establish Xid 79.
  • ✓For a rented GPU, send the provider the event time with timezone, instance/node identity, original error and any permitted device identifiers. Ask the operator for host-side evidence before requesting disruptive recovery.
  • ✓Record the operator recovery action, subsequent device/fabric state, workload results and recurrence. Absence of AER messages alone does not certify a healthy link.

Root cause

  • A PCIe link failure can make the endpoint inaccessible
  • GPU hardware failure can produce the same event
  • Driver issues can also cause Xid 79; the event is not proof of an exclusively electrical fault

The fix and how to prevent it

How to fix it

  • Preserve kernel logs, the event timestamp, GPU identity, and an NVIDIA bug report before disruptive recovery where possible.
  • Drain affected workloads and coordinate the restart with the host owner. NVIDIA's catalog lists a bare-metal host restart as the immediate action for Xid 79.
  • Use the OEM or cloud provider recovery procedure. A full power cycle may be required on some platforms, but do not claim that every warm reboot must fail.
  • Review PCIe events, platform logs, and the supported driver and firmware baseline. Escalate recurring or unresolved faults with the evidence to NVIDIA, the OEM, or the cloud provider.
  • Run approved diagnostics after draining work and verify that the GPU is usable before returning the host to service. Do not replace hardware based only on the Xid number.

Diagnose your own log automatically

The root cause and fix steps above stay free. Create a free account to analyze full job logs, attribute the first failed rank, and keep saved incident history.

Sign up free. Analyze your cluster logs

No credit card. Daily allowance follows verified trust tier. Instant access.

Why the recommended fix works

Evidence collection preserves the distinction between a device-access failure and its cause. A coordinated restart addresses the immediate host condition while subsequent diagnostics and recurrence evidence guide the vendor investigation.

Diagnostic note

“Illustrative timeline only: a host logs Xid 79 at 02:14:08 UTC and a peer reports an NCCL timeout at 02:15:08 UTC. The event order supports investigating lost device access before the peer timeout. It does not identify whether the card, link, platform or software caused it. Do not treat this illustrative timeline as a measured customer recovery.”

Xid 79 in context

Xid 79 identifies a driver-observed event. Preserve the full message, device identity and surrounding workload and host evidence before distinguishing application, driver and hardware causes. The code alone does not justify replacing a board.

Compare Xid event families

Diagnose this failure in VS Code

Select the traceback or open the failed terminal, then run Denpex locally to inspect matched failure patterns, supporting evidence and next checks without uploading the log. Verify recommendations against your workload before changing production.

Install the free VS Code extension

Frequently asked questions

Questions engineers and on-call staff commonly ask about this failure.

Does Xid 79 prove the card needs replacement?
No. The catalog includes PCIe link, device and driver explanations. Replacement needs platform and vendor investigation.
How do I diagnose this on a rented GPU?
Preserve the original job error and timestamps, identify the affected instance and ask the provider for host kernel and platform evidence. A tenant should not reset a shared host or infer a failed component from the last NCCL error.
What if nvidia-smi works on the host but fails in a container?
Investigate container device access and the runtime first. Preserve any actual host Xid events separately. A container-only NVML error is not proof that the GPU fell off PCIe.

Don't just read the fix, diagnose your run

The encyclopedia explains documented failure patterns. Denpex investigates your logs, configuration and stack, shows the supporting evidence, and asks for missing details when needed.