NVIDIA Xid error codes
An Xid is the NVIDIA kernel driver reporting that something went wrong on the GPU. The number is the whole message, it tells you whether you are looking at a bad pointer in your own code, a driver fault, or a board that needs replacing.
The distinction that costs the most money is the first one below: 10 of these 23 codes are hardware, and 10 can justify an RMA. The rest are software or driver faults, and replacing a GPU for one of those replaces a healthy card.
Every common Xid code
Sorted by code. “Class” is who is at fault in practice, not what the driver message literally says.
| Xid | Name | Class | What it means | First action |
|---|---|---|---|---|
| Xid 13 | Graphics Engine Exception | Application / software | A kernel executed an illegal operation, most often an out-of-bounds memory access or a bad pointer in application or library code. The GPU raised an exception and killed the offending context. | Run the workload under compute-sanitizer. Xid 13 is an application bug far more often than a hardware fault, and the sanitizer names the kernel and the access. |
| Xid 31 | GPU memory page fault (MMU fault) | Application / software | The GPU MMU rejected a memory access: the address was unmapped, or the access type was not permitted for that mapping. Usually a bad pointer, a use-after-free, or a race on a freed allocation. | compute-sanitizer --tool memcheck. The fault message names the engine and access type, which narrows it to a kernel quickly. |
| Xid 32 | Invalid or corrupted push buffer stream | Ambiguous, needs evidence | The command stream the driver submitted was rejected as malformed. Can be a driver/user-mode bug, and can also be memory corruption on the path that built the buffer. | Check for concurrent Xid 48/94/95 (ECC) on the same GPU, corruption upstream explains a malformed stream. Absent that, treat it as a driver/library bug and check versions. |
| Xid 38 | Driver firmware error | Driver or firmware | The driver firmware hit an internal error. Not caused by application code. | Compare driver and VBIOS against the fleet baseline; a mismatched pair is the usual cause. Reset the GPU, then update if it recurs. |
| Xid 43 | GPU stopped processing | Application / software | A channel was stopped after an error in the work it was executing. The GPU itself is healthy; the context that faulted was terminated. | Read the Xid immediately BEFORE this one, 43 is usually the cleanup that follows another fault, not the fault itself. |
| Xid 45 | Preemptive cleanup, due to previous errors | Application / software | The driver tore down a channel because something else already failed. This is TEARDOWN, not a root cause, it is also what appears when a user kills a job with Ctrl-C. | Do not diagnose Xid 45. Run `dmesg -T | grep -i xid` and read the Xid immediately BEFORE it; that one is the incident. |
| Xid 48 | Double Bit ECC Error (DBE) | HardwareRMA candidate | An uncorrectable memory error. ECC caught two flipped bits, which it can detect but not repair, so the data is lost and every context using that memory is suspect. | Drain the GPU. Resume from a checkpoint written BEFORE the Xid timestamp, anything computed after it may be silently wrong. |
| Xid 61 | Internal micro-controller breakpoint/warning | Driver or firmwareRMA candidate | The GPU management processor (PMU/falcon) hit an internal condition. The GPU is typically unusable until reset, and a repeat on one serial points at silicon or firmware. | Reset the GPU (nvidia-smi --gpu-reset -i <id>) or reboot. Check the driver/VBIOS pairing against the fleet baseline. |
| Xid 62 | Internal micro-controller halt | Driver or firmwareRMA candidate | The management processor stopped rather than merely warning. Unrecoverable without a reset, and frequently persistent. | Drain and reset. A second Xid 62 on the same serial within 30 days is an RMA with the crash dump attached. |
| Xid 63 | ECC page retirement or row remapping recording event | Hardware | The GPU retired a bad memory page (or remapped a row on A100 and later) and recorded it. This is ECC working as designed, a single event is not a failure. | Track the RATE, not the event. Check remap pool headroom: nvidia-smi -q -d ROW_REMAPPER. Retirements accelerating over days predict Xid 64. |
| Xid 64 | ECC page retirement or row remapper recording failure | HardwareRMA candidate | The GPU tried to retire a failing page or row and could not record it. The protective mechanism itself has failed, so further errors cannot be contained. | RMA. Unlike Xid 63 this is terminal, the spare-row reserve or its bookkeeping is exhausted. |
| Xid 65 | Video processor exception | Ambiguous, needs evidence | The video engine raised an exception, usually during encode or decode work. | Confirm a codec workload was running: `nvidia-smi --query-gpu=utilization.encoder,utilization.decoder --format=csv`. Isolate by disabling the hardware codec path. |
| Xid 69 | Graphics Engine class error | Application / software | An invalid class or method was submitted to the graphics engine, a driver or application-level programming error. | Run `nvidia-smi` and compare the driver/CUDA pair against the application build. This is not a hardware signal, do not drain the node for it. |
| Xid 74 | NVLink error | HardwareRMA candidate | An error on an NVLink connection between GPUs, or between a GPU and an NVSwitch. Collectives crossing the affected link either fail or fall back to a much slower path. | nvidia-smi nvlink -e for CRC/replay/recovery counters on BOTH endpoints. A counter ramp usually precedes the Xid by hours. |
| Xid 79 | GPU has fallen off the bus | HardwareRMA candidate | The driver can no longer reach the GPU over PCIe. The device stopped answering, power, thermal, link, or board fault. A warm reboot will not bring it back. | Capture dmesg, BMC event log and nvidia-bug-report.sh BEFORE power-cycling, the evidence does not survive. Then full chassis power cycle. |
| Xid 92 | High single-bit ECC error rate | Hardware | Correctable errors are being corrected, but at an elevated rate. Nothing has failed yet; this is the early-warning signal that precedes uncorrectable errors. | Schedule a drain rather than reacting immediately. Alert on the trend, a rising SBE rate is the cheapest predictor of an upcoming Xid 48/94/95. |
| Xid 94 | Contained ECC error | HardwareRMA candidate | An uncorrectable error that the GPU CONTAINED to the offending context. Other work on the GPU is unaffected; the faulting process is killed. | Kill and restart the affected job. Check row-remap headroom, containment relies on spare rows being available. |
| Xid 95 | Uncontained ECC error | HardwareRMA candidate | An uncorrectable error the GPU could NOT contain. Corruption reached wider GPU state, so all work on the device is suspect, not just the faulting process. | Drain the node. Resume from a checkpoint predating the Xid; do not trust the most recent optimizer state. |
| Xid 109 | Context switch timeout error | Ambiguous, needs evidence | The GPU did not complete a context switch within its timeout. Can be an extremely long-running kernel, and can be a driver or hardware fault. | Check for a kernel with an unbounded loop first. If the workload is normal, treat it as a driver-level fault and check the version. |
| Xid 119 | GSP RPC timeout | Driver or firmware | The driver waited for a reply from the GPU System Processor firmware and did not get one. A firmware-side stall rather than an application fault. | Reset the GPU. GSP firmware issues are strongly driver-version dependent, check NVIDIA release notes for your exact version before RMA. |
| Xid 120 | GSP error | Driver or firmware | The GPU System Processor reported an internal error. GSP runs much of the driver logic on the GPU itself from R515 onward, so a GSP fault takes the device out even though no application did anything wrong. | Reset with `nvidia-smi --gpu-reset -i <id>`, then match the driver against NVIDIA release notes. GSP can be disabled via the NVreg_EnableGpuFirmware kernel module parameter on known-affected versions. |
| Xid 140 | ECC unrecovered error | HardwareRMA candidate | An ECC error the GPU could not recover from. Treat as uncontained corruption. | Drain, resume from a checkpoint predating the event, and collect `nvidia-smi -q -d ECC,ROW_REMAPPER` output for the RMA. |
| Xid 143 | GPU initialization failure | HardwareRMA candidate | The GPU failed to initialize. It will not be usable until reset, and often not after. | Full power cycle, not a warm reboot. Then `nvidia-smi -L` to confirm enumeration. If it recurs on the same slot, test whether the fault follows the card. |
Codes not listed here exist. Rather than guess at their meaning, look them up in NVIDIA’s official Xid error reference, which is the authority for the code numbers and names above.
The three mistakes that cost the most
Diagnosing Xid 45
Xid 45 is preemptive cleanup, it fires because something else already failed, and it also fires on an ordinary Ctrl-C. The incident is whatever Xid came before it in dmesg.
RMA-ing on Xid 79 alone
“GPU has fallen off the bus” describes a power brownout, a thermal cutoff, a degraded riser and a dead board equally well. Test whether the fault follows the card into another slot before pulling it; slots that have eaten three healthy GPUs are common.
Ignoring Xid 63 and 92
Neither is a failure, which is why they get filtered out. They are the only early warning you get: a rising correctable-error rate or an accelerating page-retirement count reliably precedes the uncontained errors that kill a run.
Frequently asked questions
Is an Xid error always a hardware failure?
No, and assuming so is the most expensive mistake made with these codes. Xid 13, 31, 43 and 69 are overwhelmingly application bugs, a bad pointer or an out-of-bounds access. Xid 48, 94, 95, 64 and 79 point at the board or its environment. Xid 45 is neither: it is teardown that follows some other fault, and it also appears when a user simply cancels a job.
Which Xid codes justify an RMA?
Uncontained or unrecoverable memory errors (48, 94, 95, 140), a failed page-retirement or row-remap record (64), NVLink faults that survive a reseat (74), repeated micro-controller halts (61, 62), and a GPU that keeps falling off the bus after the node's power and cooling have been cleared (79). A single occurrence of a correctable-class event such as 63 or 92 is not an RMA, it is a signal to start watching the trend.
Where do I find Xid messages?
In the kernel log, prefixed NVRM. Run `dmesg -T | grep -i xid` on the host, or check /var/log/kern.log. They are host-side kernel events, so they do not appear in a container's logs even when the container's process is the one that died.
Why did several GPUs report an Xid at the same instant?
Independent GPUs do not fail simultaneously by chance. Simultaneous Xids across a node almost always mean shared infrastructure, power delivery on the baseboard, cooling, or the PCIe/NVSwitch fabric. Investigate the node, not the cards; replacing a GPU will not fix it.
Does a reboot clear an Xid?
It clears the driver state, not the cause. For Xid 79 specifically a warm reboot does not even do that, the device is off the PCIe bus and only a full power cycle re-enumerates it. Always capture dmesg, the BMC event log and nvidia-bug-report.sh output before cycling, because none of it survives.
Reading an Xid is the easy part
The hard part is which rank failed first, whether the checkpoint is safe, and whether the card or the slot is at fault. Paste your dmesg and training log together and Denpex answers all three.
Diagnose your logs free