Operator task guide

How to determine whether a GPU needs RMA

A GPU needs RMA when repeatable, device-owned hardware evidence survives reasonable software, node and slot controls. One ambiguous Xid, CUDA crash or missing device is not enough because power, cooling, PCIe, driver and application faults can create the same outer symptom.

Preserve device identity and host evidence before resetting, classify the Xid or ECC event, check recurrence on the same serial, and run the cheapest test that separates the card from its environment. Then use one of three honest outcomes: RMA candidate, field diagnosis first, or not RMA-worthy.

Step-by-step method

1
Preserve evidence before reset or power cycle
Kernel, BMC and driver evidence can disappear during recovery. Record the GPU UUID, serial, PCI bus ID, node, timestamp and every simultaneous event before touching the device.
```
dmesg -T | rg -i 'nvrm|xid|ecc|nvlink|pcie'
nvidia-bug-report.sh
nvidia-smi --query-gpu=index,uuid,serial,pci.bus_id,driver_version,vbios_version --format=csv
```
2
Classify the initiating hardware signal
Uncorrectable ECC, failed row remapping, repeated microcontroller halts and persistent link loss carry more hardware weight than application-class Xids or a generic CUDA error. Read the event in its full sequence, not as an isolated number.
3
Rule out software ownership
Reproduce the workload with a known-good driver and image or with a sanitizer when the Xid class points to application memory access. A failure that disappears under serialization or follows the input is not strong board evidence.
4
Check recurrence by serial and shared-node events
Repeated events on one serial strengthen card ownership. Simultaneous events across several GPUs weaken it and point toward shared power, cooling, baseboard, PCIe or NVSwitch infrastructure.
```
nvidia-smi -q -d ECC,ROW_REMAPPER,PAGE_RETIREMENT
nvidia-smi nvlink -e
```
5
Run the card-versus-slot control
When operational policy permits, move the suspect card to a known-good slot and put a known-good card in the suspect path. A fault that follows the card supports RMA; one that remains with the slot or node does not.
6
Package the verdict and missing evidence
State the verdict, causal evidence, controlled test, recurrence, device identity, commands run and anything still missing. Do not turn an unresolved case into a hardware claim to make the ticket look complete.

RMA decision signals

Signals, meanings and actions for how to determine whether a gpu needs rma.
Signal	What it means	Next action
Uncorrectable ECC or failed row remapping that recurs on one serial	Strong device-memory evidence	Keep drained, protect checkpoints and prepare the vendor packet.
Xid 79, GPU fallen off the bus	Device communication is gone, but ownership is still ambiguous	Check BMC power, thermal and PCIe evidence, then test whether the failure follows the card.
Xid 13, 31, 43 or device-side assert tied to one workload	Application or software evidence	Use sanitizer or request controls. Do not RMA from this signal alone.
Several GPUs fail at the same timestamp	Shared node or fabric cause is more likely than independent cards	Investigate power, baseboard, cooling, PCIe and NVSwitch before replacing boards.
Fault stays with slot after a card swap	The card is not the recurring owner	Repair the slot, riser, node or shared path and keep the healthy board out of the RMA queue.

Evidence checklist

GPU UUID, serial, PCI bus ID, VBIOS and driver version
Complete Xid sequence and kernel context before reset
ECC, row-remapper, page-retirement and NVLink counters
BMC power, thermal and chassis events
Node topology and simultaneous peer-GPU failures
Controlled card, slot, node or workload comparison

Common mistakes

Treating Xid 79 as a dead board

The driver lost the device, but power, thermal, riser, slot, baseboard and card faults all satisfy that description.

Resetting before collection

A successful reset can erase the only device and host evidence the vendor needs, without proving the cause was fixed.

Using GPU index as identity

Indexes change across boots, containers and scheduler steps. Correlate recurrence by UUID or serial, with PCI bus and host as location evidence.

Tools and deeper references

NVIDIA Xid decoder

Paste the code and review fault class, RMA relevance and first action.

nvidia-smi parser

Extract identity, memory, thermal, power and ECC evidence in browser.

Xid 79 investigation

Follow the complete GPU-fallen-off-bus decision path.

OEM and RMA workflows

Connect field diagnosis, vendor evidence and outcome measurement.

Official references

Frequently asked questions

Does one Xid 79 mean the GPU needs RMA?

No. Xid 79 means the driver can no longer communicate with the GPU. A dead board is one cause, but power, thermal, riser, slot, baseboard and PCIe failures can look identical. Preserve evidence and test whether the fault follows the card.

Which signals are strongest for GPU RMA?

Recurring uncorrectable memory errors, exhausted or failed row remapping, persistent microcontroller failures, and link or initialization faults that follow one serial after node and software controls provide the strongest operational evidence.

Can a GPU pass nvidia-smi and still be bad?

Yes. nvidia-smi proves the driver can query the device at that moment. Intermittent memory, PCIe, NVLink, power or thermal failures may disappear at idle. Use recurrence plus a production-shaped health control and preserve the prior fault evidence.

Who makes the final warranty decision?

The OEM or vendor makes the warranty decision under its support terms. Denpex can classify and package operational evidence, mark missing fields and screen out not-RMA-worthy cases, but it does not certify a board as defective.

Apply the method to the complete incident

Paste the logs for an answer-first diagnosis, evidence request and verification step.

Diagnose the incident