Xid 79: GPU has fallen off the bus

Xid 79 means the driver can no longer reach the GPU over PCIe. The device stopped responding to config-space reads mid-operation, so the kernel dropped it. Every process holding a CUDA context on that GPU dies, and nvidia-smi can no longer enumerate it until the host is power-cycled.

Quick answer

Xid 79 means the GPU stopped responding on the PCIe bus and the driver lost it entirely. It is a hardware, power, or thermal fault far more often than a software bug, a warm reboot will not recover it, and the node needs a full power cycle before the GPU re-enumerates.

Hardware#xid#xid-79#pcie#nvidia-driver#hardware#rma

What this failure is

Xid 79 is an NVRM error the NVIDIA kernel driver logs when it issues a PCIe transaction to a GPU and the device does not answer. The driver reads back all-ones (0xFFFFFFFF) from config space, concludes the endpoint is gone, and marks the GPU lost. Unlike most Xids, which report a fault inside a running kernel, Xid 79 reports that the GPU itself has disappeared from the host's point of view.

Live diagnosis, no signup

Is this what broke your run? Paste your log.

You're reading about Xid 79: GPU has fallen off the bus. Paste your own crash log or traceback below and get the real root cause for YOUR run, not this generic entry. No account, no card. Logs are masked at ingress and never saved to account history.

training_logs.txt

No log to hand? Try one:

3 free diagnoses/day

Want 14 days of full access?

Request a free trial code for unlimited diagnoses, alerts, history, and follow-up questions. No credit card.

Request 14-day trial

Why it happens (the mechanism)

A PCIe endpoint disappears when it stops driving the link. That happens for a small number of physical reasons: the GPU lost power or browned out under a transient load spike, it overheated and the hardware protection cut it off, the physical link degraded until training failed, or the board has a genuine fault in the voltage regulators or the PCIe interface. On SXM baseboards the same symptom appears when a board-level power rail sags under simultaneous all-GPU load. Because the failure is electrical, the driver has no way to recover it in software, the device is not there to be reset.

What you'll observe

Training or inference job dies instantly with a CUDA error and cannot be restarted on the same node
nvidia-smi no longer lists the GPU, or lists it and returns Unknown Error for that device
A warm reboot does not bring the GPU back; only a full power cycle re-enumerates it
The same GPU or the same physical slot fails repeatedly, often under heavy sustained load

Common symptoms and what they mean

Symptom	Why it happens
dmesg contains NVRM: Xid (PCI:0000:XX:00): 79, GPU has fallen off the bus	Power delivery failure: a transient current spike during a synchronised all-GPU workload pulls a rail below spec, and the GPU drops off the bus before the PSU can respond. This is the most common cause on dense nodes and is why the failure often hits several GPUs at once.
nvidia-smi returns 'Unable to determine the device handle for GPU 0000:XX:00.0: Unknown Error'	Thermal shutdown: inlet temperature, a failed fan, or blocked airflow drives the GPU past its slowdown threshold into hardware protection, cutting the device off the link.
lspci shows the device present but with all-ones config space, or the device is missing entirely	Physical PCIe link degradation: a marginal riser, cable, connector, or reseated card causes the link to retrain repeatedly and eventually fail. Frequently correlates with correctable PCIe AER errors logged before the Xid.
The GPU count reported by nvidia-smi drops below the expected count for the chassis	Genuine board failure: a fault in the GPU's voltage regulators or PCIe interface. This is the case that requires RMA, and it is a minority of Xid 79 events despite being the first assumption.
Failures cluster on one PCIe slot or one SXM baseboard position rather than following the GPU when it is moved	On virtualised or passthrough hosts, an IOMMU or hypervisor reset of the device can present identically to a hardware disappearance.

Which systems are affected

NVIDIA data-center GPUs (A100, H100, H200, L40S, B200) in both PCIe and SXM form factors
Multi-tenant bare-metal GPU nodes running sustained high-power workloads
HGX baseboards where several GPUs share power delivery and cooling
Any host where PSU headroom was sized for average rather than peak transient draw

How to confirm this is the problem

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

✓dmesg or the kernel log contains the literal string 'Xid' followed by ', 79,' and 'GPU has fallen off the bus'
✓nvidia-smi either omits the GPU or returns 'Unknown Error' for that specific device index
✓The GPU does not return after `nvidia-smi -r` or a warm reboot, but does return after a full chassis power cycle
✓lspci -vvv for the device shows all-ones config space, or the device no longer appears in lspci output
✓No CUDA process can create a context on that device; every existing context died at the same instant

Example training logs (fingerprint)

training.log (synthetic fingerprint)

[Sat Aug 22 03:14:52 2026] NVRM: Xid (PCI:0000:8a:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[Sat Aug 22 03:14:52 2026] NVRM: GPU 0000:8a:00.0: GPU has fallen off the bus.
[Sat Aug 22 03:14:52 2026] NVRM: A GPU crash dump has been created. If possible, please run nvidia-bug-report.sh as root to collect this data.
[Sat Aug 22 03:14:53 2026] pcieport 0000:88:02.0: AER: Uncorrected (Non-Fatal) error received: 0000:8a:00.0
$ nvidia-smi
Unable to determine the device handle for GPU 0000:8A:00.0: Unknown Error

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

The fix and the prevention pattern

The root cause is on this page and stays free. A free account adds the exact remediation steps, keeps your diagnoses instead of discarding them, and unlocks the fix on every entry in the encyclopedia.

No credit card · 3 free diagnoses · Instant access

Why the recommended fix works

The distinction that matters is whether the fault follows the card or stays with the slot, because those two answers lead to opposite actions. Xid 79 is treated as a dead GPU by default, and that assumption sends healthy cards back under warranty while the node that actually killed them stays in the pool and kills the replacement. Reading the BMC power and thermal logs for the same timestamp, and checking for PCIe AER errors leading up to the event, separates a power or cooling problem from a board fault before any hardware is moved. The swap test settles it definitively. Power-cycling is required first in every case simply because a device that is off the bus cannot be reset in software.

Code examples

snippet

# Capture the evidence BEFORE the node is power-cycled, a reboot destroys it.
# Run this from the host as soon as an Xid 79 is detected.

set -euo pipefail
OUT="/var/log/denpex/xid79-$(date +%Y%m%dT%H%M%S)"
mkdir -p "$OUT"

# 1. The Xid itself plus surrounding PCIe events.
dmesg -T | grep -i -E 'xid|pcieport|AER|NVRM' | tail -200 > "$OUT/dmesg.txt"

# 2. Which device the driver can and cannot see.
nvidia-smi -L                                  > "$OUT/nvidia-smi-L.txt" 2>&1 || true
nvidia-smi --query-gpu=index,serial,pci.bus_id,temperature.gpu,power.draw \
           --format=csv                        > "$OUT/nvidia-smi-q.txt" 2>&1 || true

# 3. PCIe state. All-ones config space confirms the endpoint is gone.
lspci -nn | grep -i nvidia                     > "$OUT/lspci.txt" 2>&1 || true

# 4. Platform-side cause: power and thermal events at the same timestamp.
ipmitool sel elist | tail -50                  > "$OUT/bmc-sel.txt" 2>&1 || true

# 5. The full NVIDIA bug report, which contains the GPU crash dump.
nvidia-bug-report.sh --output-file "$OUT/nvidia-bug-report.log.gz" || true

echo "Evidence written to $OUT, safe to power cycle now."

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.

Best practices by model family

Model / Stack	Recommendation	Notes
Single PCIe GPU workstation	Check PSU and riser first	Consumer and workstation PSUs are the most common cause here; the card is usually fine.
HGX / SXM 8-GPU node	Check baseboard power and BMC log first	Shared power delivery means a rail sag can drop several GPUs at once. Simultaneous Xid 79 on multiple GPUs is a node fault, never a coincidence of card failures.
Multi-tenant rented node	Capture evidence before recycling	The dmesg window and BMC log do not survive the power cycle, and without them an RMA cannot be justified.
Virtualised / GPU passthrough	Rule out hypervisor reset first	An IOMMU or VFIO device reset presents identically from inside the guest. Check the host before blaming hardware.
Long training run (>24h)	Treat as a node eviction	The GPU will not return without a power cycle, so the job must checkpoint and reschedule rather than retry in place.

With the fix vs without the fix

Dimension	With the fix	Without the fix
Recovery without power cycle	Not possible, device is off the bus	Repeated warm reboots that never restore the GPU
Correct disposition of the hardware	Slot-vs-card swap test identifies the true fault	Healthy cards RMA'd while the faulty node keeps killing replacements
Evidence available for a warranty claim	dmesg, BMC SEL and bug report captured pre-reboot	Evidence destroyed by the power cycle; claim rejected
Repeat rate on power-marginal nodes	Reduced by power capping and PSU headroom	Recurs under every synchronised peak load
Cost of a single event to a training run	Minutes, from the last checkpoint	Hours of recomputation, repeated per occurrence

Real engineering notes

“Xid 79 is the error most often mistaken for a dead GPU. The driver message is unambiguous that the device is gone, so the card gets pulled, but 'the GPU stopped answering' describes a power brownout, a thermal cutoff, and a failed voltage regulator equally well, and only the last one is the card's fault. The cheapest discriminator is whether the failure follows the card into a different slot. Fleets that skip that test show a characteristic pattern: a slot that has consumed three GPUs, all three of which tested fine at the vendor.”

Visual fingerprint

Deciding whether the card or the node is at fault

  Xid 79 logged
       |
       v
  Capture dmesg + BMC SEL + bug report   <- do this BEFORE power cycling
       |
       v
  Multiple GPUs down at the same instant? --yes--> node power delivery, not the cards
       | no
       v
  BMC shows over-temp or over-current?   --yes--> cooling / power, not the card
       | no
       v
  PCIe AER errors before the Xid?        --yes--> link: riser, cable, slot
       | no
       v
  Move card to a different slot, stress test
       |
       +-- fails again in new slot ------> RMA the card
       +-- new card fails in old slot ---> RMA nothing, fix the node

The swap test is the only step that distinguishes a faulty board from a faulty slot. Everything above it is cheaper and rules out the more common causes first.

Xid 79 in context

Xid 79 is one of a small set of codes the NVIDIA driver uses to report GPU faults, and the number is most of the diagnosis: it tells you whether you are looking at your own code, the driver, or a board that needs replacing.

Compare every Xid code side by side

Diagnose this failure in VS Code

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

Install the free VS Code extension

Root cause

Power delivery failure: a transient current spike during a synchronised all-GPU workload pulls a rail below spec, and the GPU drops off the bus before the PSU can respond. This is the most common cause on dense nodes and is why the failure often hits several GPUs at once.
Thermal shutdown: inlet temperature, a failed fan, or blocked airflow drives the GPU past its slowdown threshold into hardware protection, cutting the device off the link.
Physical PCIe link degradation: a marginal riser, cable, connector, or reseated card causes the link to retrain repeatedly and eventually fail. Frequently correlates with correctable PCIe AER errors logged before the Xid.
Genuine board failure: a fault in the GPU's voltage regulators or PCIe interface. This is the case that requires RMA, and it is a minority of Xid 79 events despite being the first assumption.
On virtualised or passthrough hosts, an IOMMU or hypervisor reset of the device can present identically to a hardware disappearance.

The fix and how to prevent it

Unlock the full remediation runbook

14 days on the Scale plan, up to 50 diagnoses a day. Step-by-step remediation, the RMA evidence payload, and multi-node correlation on your own logs. No card, and it does not roll into a subscription.

Frequently asked questions

Twelve targeted questions that engineers and on-call staff most commonly ask about this failure.

Does Xid 79 always mean the GPU is dead?

No, and this is the most expensive misconception about it. Xid 79 states that the driver lost contact with the device over PCIe. A power brownout, a thermal cutoff, a degraded riser and a genuinely failed board all produce that same message. A real board failure is a minority of cases, so pulling the card should be the last step, not the first.

Why doesn't a reboot bring the GPU back?

A warm reboot does not remove power from the PCIe slot, so a device that stopped driving the link stays gone. The host has to fully lose power for the endpoint to re-enumerate. On a remote node, use the BMC to power-cycle the chassis rather than issuing `reboot`.

Several GPUs on the node dropped at once. What does that mean?

It means the node, not the cards. Independent GPUs do not fail simultaneously by chance. Shared power delivery on an HGX baseboard sagging under a synchronised load is the usual cause, and the BMC event log for that timestamp will normally confirm it.

Can a software bug cause Xid 79?

Not directly. The GPU is unreachable at the PCIe level, which software on the host cannot cause. The exception is virtualised hosts, where a hypervisor or IOMMU device reset can look identical from inside the guest, so on passthrough setups, check the host before concluding hardware.

How do I justify an RMA for this?

Capture the dmesg window containing the Xid, the BMC event log showing no power or thermal event, the PCIe AER history, and `nvidia-bug-report.sh` output, all from before the power cycle. Then show the fault follows the card into a different slot. Without the swap test, vendors frequently return the card as no-fault-found.

Will power capping stop it?

On power-marginal nodes, often yes. `nvidia-smi -pl <watts>` limits the transient draw that causes rails to sag during synchronised collectives. It costs some throughput, so it is a mitigation for a node you know is marginal rather than a default.

Is Xid 79 related to Xid 48 or Xid 92?

They are different failure classes. Xid 48 and 92 are memory errors reported by a GPU that is still present and responding. Xid 79 means the GPU is no longer there to report anything. A GPU can log ECC errors for weeks before an unrelated power fault takes it off the bus.

What should a training job do when this happens?

Treat it as a node eviction, not a retryable error. The GPU will not return without a power cycle, so retrying in place always fails. Elastic training with a rendezvous backend should drop the node and continue on the remaining workers from the last checkpoint.

References

Don't just read the fix, diagnose your run

The encyclopedia tells you what went wrong. Denpex tells you what went wrong in YOUR training run. With your logs, your config, and your stack.

Diagnose your logs free Browse all failures

Related Hardware errors

GPU Memory Clock Throttle

Hardware · medium

NVLink Error

Hardware · high

GPU Thermal Throttling

Hardware · high

GPU Power Cap Reached

Hardware · medium