CUDA uncorrectable ECC error on an NVIDIA GPU

An uncorrectable GPU memory error means ECC could not repair the affected data. On modern NVIDIA data-center GPUs, the next action depends on whether the error was contained, the recovery-action flag, row-remapping state, and field diagnostics. A single message does not automatically prove the board needs RMA.

An uncorrectable ECC error is a memory error that the GPU's ECC mechanism cannot correct. NVIDIA reports related state through Xid events, NVML or nvidia-smi, row-remapping fields, and recovery-action flags. Xid 48 reports an uncorrectable ECC event, while Xids 94 and 95 distinguish contained and uncontained outcomes on supported products.

A memory or cache location returned data outside ECC's correction capability. The event can be isolated by containment, can require a GPU reset so pending row remapping takes effect, or can expose a persistent condition that meets NVIDIA's RMA policy after field validation. The user-visible CUDA exception is the workload impact, not the complete hardware disposition.

• The affected application can terminate or report corrupted execution state.
• An uncontained event can affect more than the process that encountered the faulty location.
• Resetting before evidence capture can erase the incident context needed for support and RMA review.
• Automatically replacing every GPU after one uncorrectable event can create false RMAs.

• NVIDIA data-center GPUs with ECC-enabled memory
• Multi-tenant or MIG systems where containment determines blast radius
• Schedulers that need a drain, reset, validation, and return-to-service workflow

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Find the first Xid and CUDA error for the affected GPU and map its PCI bus ID or UUID to the job and rank.
• ✓Determine whether Xid 94 or Xid 95, or the corresponding recovery-action state, reports a contained or uncontained error.
• ✓Capture full nvidia-smi -q output before reset, including ECC and row-remapping fields exposed by the device.
• ✓Follow the required reset or recovery action, then run the platform-approved GPU diagnostic.
• ✓Return the GPU to service only after the same production-shaped control passes and no new uncorrectable event appears.

NVRM: Xid (PCI:0000:01:00): 95, Uncontained

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Containment, recovery, and RMA are different decisions. Preserving the Xid and device state identifies the blast radius. Following the recovery-action and row-remapping state restores the hardware at the required boundary. Field diagnostics prevent an unsupported replacement decision based only on a wrapper exception.

# Preserve evidence before a reset
dmesg -T | grep -iE 'NVRM|Xid|ECC'
nvidia-smi -q
nvidia-smi --query-gpu=timestamp,uuid,pci.bus_id,ecc.errors.uncorrected.volatile.total,ecc.errors.uncorrected.aggregate.total --format=csv

# Scheduler containment example
scontrol update NodeName=<node> State=DRAIN Reason='uncorrectable ECC investigation'

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.

Xid 48 is one of a small set of codes the NVIDIA driver uses to report GPU faults, and the number is most of the diagnosis: it tells you whether you are looking at your own code, the driver, or a board that needs replacing.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• ECC detected a memory result that could not be corrected.
• The physical owner and required action depend on the reported location, containment state, recovery-action flag, and persistent row-remapping or SRAM threshold state.
• The event becomes an RMA case only when NVIDIA's policy condition is met and validated by the field diagnostic process.