NVIDIA Xid 79. GPU Has Fallen Off the Bus
Xid 79 means the GPU lost PCIe connectivity entirely. Training dies immediately. Denpex identifies which GPU, node, and the chain of events that led to the GPU fall. Root caused by bad PCIe, power events, or driver timeouts.
Xid 79 means the GPU lost PCIe connectivity entirely.
What this failure is
NVIDIA Xid 79. GPU Has Fallen Off the Bus is a Hardware failure seen during ML training runs. Xid 79 means the GPU lost PCIe connectivity entirely. Training dies immediately. Denpex identifies which GPU, node, and the chain of events that led to the GPU fall. Root caused by bad PCIe, power events, or driver timeouts. Common tags: Xid 79, Pcie, Gpu, Bus.
Is this what broke your run? Paste your log.
You're reading about NVIDIA Xid 79. GPU Has Fallen Off the Bus. Paste your own crash log or traceback below and get the real root cause for YOUR run, not this generic entry. No account, no card. Logs are masked at ingress and never saved to account history.
Want 14 days on the Scale plan?
Request an evaluation code. A verified workplace organization activates up to 50 diagnoses a day, alerts, history, and follow-up questions. No credit card or automatic subscription.
Why it happens (the mechanism)
GPU lost PCIe link: loose cable, power event, or thermal throttling causing PCIe link retraining. Driver timeout waiting for GPU to respond: CUDA_ERROR_DEINITIALIZED. PCIe slot power delivery failure: GPU resets itself and training processes lose their CUDA context. Thermal shutdown: GPU cooler failure, throttling below usable state. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.
What you'll observe
- Training dies immediately with no graceful shutdown
- All CUDA operations on the GPU fail instantly
- nvidia-smi shows the GPU as unavailable or unreachable
Common symptoms and what they mean
| Symptom | Why it happens |
|---|---|
| dmesg: GPU has fallen off the bus on GPU X | GPU lost PCIe link: loose cable, power event, or thermal throttling causing PCIe link retraining |
| nvidia-smi returns 'No devices found' for the affected GPU | Driver timeout waiting for GPU to respond: CUDA_ERROR_DEINITIALIZED |
| Job crashes with CUDA_ERROR_INVALID_PCI something to GPU X | PCIe slot power delivery failure: GPU resets itself and training processes lose their CUDA context |
Which systems are affected
- Bare-metal GPU servers
- DGX systems with PCIe riser cards
- Power cycling GPUs
- Systems with loose PCIe power cables
- Bare-metal clusters with thermal events
How to confirm this is the problem
Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.
- ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
- ✓Verified signal present: dmesg: GPU has fallen off the bus on GPU X
- ✓Verified signal present: nvidia-smi returns 'No devices found' for the affected GPU
- ✓Verified signal present: Job crashes with CUDA_ERROR_INVALID_PCI something to GPU X
- ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.
The fix and the prevention pattern
The root cause is on this page and stays free. A free account adds the exact remediation steps, saved history, and the fix on every entry in the encyclopedia.
Sign up free. Unlock the full analysisNo credit card. Daily allowance follows verified trust tier. Instant access.
Xid 79 in context
Xid 79 is one of a small set of codes the NVIDIA driver uses to report GPU faults, and the number is most of the diagnosis: it tells you whether you are looking at your own code, the driver, or a board that needs replacing.
Compare every Xid code side by sideDiagnose this failure in VS Code
Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.
Install the free VS Code extensionRelated failures to investigate next
Root cause
- GPU lost PCIe link: loose cable, power event, or thermal throttling causing PCIe link retraining
- Driver timeout waiting for GPU to respond: CUDA_ERROR_DEINITIALIZED
- PCIe slot power delivery failure: GPU resets itself and training processes lose their CUDA context
- Thermal shutdown: GPU cooler failure, throttling below usable state
The fix and how to prevent it
Evaluate Denpex on your own logs
Request a Scale evaluation code. A verified workplace organization activates 14 days with up to 50 diagnoses a day. Every account keeps its current diagnosis allowance and gets a verification path. No card or automatic subscription.
Don't just read the fix, diagnose your run
The encyclopedia tells you what went wrong. Denpex tells you what went wrong in YOUR training run. With your logs, your config, and your stack.