Xid 92: High single-bit ECC error rate

Xid 92 reports a high rate of corrected single-bit ECC errors. Corrected errors preserve the original data; this event does not prove uncorrectable memory errors, corrupted tensors, or the cause of NaN loss.

Symptom

Xid 92

Root cause

Xid 92 reports a high rate of corrected single-bit ECC errors. Corrected errors preserve the original data; this event does not prove uncorrectable memory errors, corrupted tensors, or the cause of NaN loss.

Recommended fix

Record volatile and aggregate ECC counters, timestamps, GPU UUID, and row-remapping status. Compare counter deltas over time and schedule vendor-supported diagnostics after draining work. Investigate NaNs independently rather than attributing them to corrected bits.

How Denpex helps

Denpex investigates Xid 92: High single-bit ECC error rate using the evidence you provide or your connected workload collects. Earlier rank, host or application evidence is needed to distinguish an initiating failure from a downstream report.

Xid 92 reports a high rate of corrected single-bit ECC errors. Corrected errors preserve the original data; this event does not prove uncorrectable memory errors, corrupted tensors, or the cause of NaN loss.

Xid 92 reports a high rate of corrected single-bit ECC errors. Corrected errors preserve the original data; this event does not prove uncorrectable memory errors, corrupted tensors, or the cause of NaN loss. Use incident telemetry and the supported driver/firmware baseline to investigate the underlying mechanism.

• Xid 92 reports a high rate of corrected single-bit ECC errors. Corrected errors preserve the original data; this event does not prove uncorrectable memory errors, corrupted tensors, or the cause of NaN loss.
• The event requires investigation; it does not establish every proposed downstream effect.

• NVIDIA data center GPUs and the Linux driver
• production-shaped multi-accelerator workloads
• containerized and bare-metal deployments of the same stack

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Preserve the literal Xid code and incident timestamp from kernel logs.
• ✓Record nvidia-smi -q output, including GPU UUID, ECC and row-remapping details.
• ✓Compare fresh counter deltas and diagnostics with the fleet baseline.

• Xid 92 reports a high rate of corrected single-bit ECC errors. Corrected errors preserve the original data; this event does not prove uncorrectable memory errors, corrupted tensors, or the cause of NaN loss.

Xid 92

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Trend monitoring and diagnostics distinguish a transient correction from persistent degradation without assuming data corruption.

# Read-only evidence collection
dmesg -T | grep -iE "NVRM|Xid|ECC"
nvidia-smi -q -d ECC,ROW_REMAPPER
# Preserve a GPU bug report before reset or reboot
sudo nvidia-bug-report.sh

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.

Xid 92 is one of a small set of codes the NVIDIA driver uses to report GPU faults, and the number is most of the diagnosis: it tells you whether you are looking at your own code, the driver, or a board that needs replacing.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.