Hardware error references

Browse 86 published references for hardware failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• Coolant Distribution Unit CDU secondary loop flow rate low immediate system power-off

  Liquid cooling flow dropped below the safe threshold. This is a facilities fault with a very short fuse, direct-liquid-cooled GPUs have little thermal mass, so an emergency power-off follows quickly and every job on the affected loop dies at once. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent blackwell failures.

• CUDA Device Assertion Failure

  CUDA device assertion failures indicate GPU hardware problems that crash training with ambiguous error messages. Denpex maps the device assertion to the specific GPU and operation, distinguishing hardware faults from driver issues.

• CUDA Driver Crash

  CUDA driver crash terminates all GPU processes on the node, losing unsaved training progress.

• CUDA Illegal Instruction

  CUDA illegal instruction errors crash training when the GPU encounters unsupported instructions, often from binary mismatches.

• CUDA uncorrectable ECC error on an NVIDIA GPU

  An uncorrectable GPU memory error means ECC could not repair the affected data. On modern NVIDIA data-center GPUs, the next action depends on whether the error was contained, the recovery-action flag, row-remapping state, and field diagnostics. A single message does not automatically prove the board needs RMA.

• cuMemImportFromShareableHandle Fails (CUDA error 101)

  P2P fabric-handle import between containers on a multi-node NVLink (MNNVL) GPU fails with CUDA 'invalid device ordinal' due to a CUDA driver bug, not NCCL.

• CUTLASS SM120 Kernel Regression Halving MoE Throughput

  NVIDIA's CUTLASS kernels for SM120 (and some Hopper architectures) suffer from a silent regression that halves throughput on Mixture of Experts (MoE) inference, commonly seen when utilizing certain PyTorch extensions or vLLM.

• DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR counter incremented link degraded

  DCGM observed an increase in NVLink CRC flit errors and marked the link degraded. A rising counter indicates signal or link integrity trouble, not a collective software timeout. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent nvlink failures.

• DCGM_FI_DEV_ROW_REMAP_FAILURE pending row remap failed bank row reserve exhausted

  HBM row remapping failed: the GPU has exhausted its reserve of spare memory rows and can no longer retire failing ones. This is a terminal hardware state, the GPU will keep producing uncorrectable ECC errors and must be replaced. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent telemetry failures.

• die-to-die thermal delta between primary die and secondary die emergency throttle

  A Blackwell B200 module detected an unsafe temperature difference between its two dies and entered emergency throttling. The delta points to uneven cooling or contact, not ordinary workload-wide thermal saturation. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent blackwell failures.

• Fabric Manager initialization failed: NVLink topology match failed

  NVIDIA Fabric Manager failed to initialize, so NVSwitch routing was never programmed. On an HGX/DGX baseboard every NVLink-based collective is unavailable until this is fixed, jobs either fall back to PCIe and crawl, or fail outright when they request peer memory. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent nvlink failures.

• Fabric-Induced SM Containment Cascade

  A transient transmission failure on the NVLink or RoCE fabric propagates corrupted state to the SM. The GPU isolates the SM fault to prevent data corruption, emitting a containment Xid without an actual memory bit flip.

• FlashAttention Unsupported GPU Architecture

  FlashAttention v2 heavily optimizes memory access and matrix multiplication using specific hardware features introduced in NVIDIA's Ampere architecture (Compute Capability 8.0+), such as asynchronous memory copies and specific Tensor Core instructions. Older architectures (Volta, Turing) lack these hardware features, making the optimized CUDA kernels impossible to run.

• GPU Clock Throttle

  GPU clock throttle reduces performance when power or thermal limits are reached.

• GPU Fallen Off Bus (Xid 79) causing Async NCCL Hang

  A severe hardware fault (often power delivery or PCIe signal integrity) causes the GPU to disconnect from the PCIe bus entirely. The CUDA runtime loses contact with the device, and NCCL operations immediately fail, cascading into timeouts across other nodes waiting for this GPU.

• GPU Fallen Off the Bus (Xid 79)

  A GPU drops off the PCIe bus (Xid 79) and disappears from the system. A critical hardware/connectivity fault that ends training on that node until the GPU is recovered.

• GPU Memory Bandwidth Limit

  GPU memory bandwidth limits reduce training performance when compute exceeds memory access speed.

• GPU Memory Bus Error

  GPU memory bus errors corrupt data transfer between GPU cores and memory, causing silent data corruption or crashes.

• GPU Memory Clock Throttle

  GPU memory clock throttling reduces memory bandwidth and training performance when memory is under-utilized.

• GPU Memory Row Remapping Event

  The GPU remaps a failing memory row after ECC errors. A recovery mechanism, but a rising remap count signals degrading memory that may need RMA.

• GPU MIG (Multi-Instance GPU) Mode

  GPU MIG mode splits a GPU into multiple instances, which can cause issues with PyTorch and NCCL if not configured correctly.

• GPU Not Detected

  GPU not detected errors prevent training from starting when the system cannot see the GPU.

• GPU Overheating

  GPU overheating causes thermal throttling, performance degradation, and potential hardware damage.

• GPU Power Cap Reached

  GPU power cap limits GPU power consumption, reducing performance for power-constrained deployments.

• GPU Thermal Slowdown or Power Brake? Check the Reason

  Distinguish SW thermal slowdown, HW thermal slowdown and external power brake before changing cooling, power settings or the workload.

• GPU Thermal Throttling

  GPU thermal throttling reduces clock speeds to prevent overheating, causing training to be slower than expected.

• GSP Firmware Silent Death (Bus Drop)

  The GPU System Processor (GSP) firmware crashes internally, terminating the PCIe link negotiation. This can occur even under near-zero utilization or idle states.

• GSP RPC Timeout (Xid 119)

  The GPU System Processor (GSP) stops responding to RPCs, logged as Xid 119. The GPU becomes unresponsive and usually needs a reset.

• habana synapseai HCLError HCL_TIMEOUT waiting for AllReduce on Gaudi OAM module

  A Habana collective timed out waiting for a peer. Like every collective timeout this names the ranks that WERE waiting, not the one that failed to arrive, the OAM module in the message is the reporter, not necessarily the fault. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent non-nvidia failures.

• HBM Silent Data Corruption leading to Loss Spikes

  A transient hardware fault occurred in an unprotected part of the GPU memory pipeline (e.g., a multi-bit flip that bypassed ECC, or a fault in the L1 cache/SRAM). This resulted in an incorrect matrix multiplication output (e.g., a massive floating-point value). Because it wasn't caught by hardware parity checks, the bad data silently poisoned the gradients.

• High ECC Error Rate Detection

  High GPU ECC error rates indicate degrading memory hardware. Correctable errors accumulating signal impending failure.

• HIP error: hipErrorOutOfMemory Out of device memory AMD Instinct /dev/kfd

  HIP runtime returned Out of Memory error. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent non-nvidia failures.

• HW Power Brake Asserted GPU power capped check chassis PSU redundancy

  SXM power brake event. GPU exceeded power budget; system forced power limiting. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent telemetry failures.

• InfiniBand Link Flap Triggering NCCL Async Handshake Failure

  A brief hardware network event (link flap, switch reset, optical transceiver issue) breaks the reliable InfiniBand (RC) connection between nodes. NCCL encounters an unrecoverable RDMA error and raises it, triggering the PyTorch watchdog to cleanly tear down the job.

• Intel GPU (i915/Xe) Engine Hangs and GuC Firmware Failures

  On Intel data-center GPUs, engine hangs surface as i915 "GPU HANG"/engine-reset dmesg lines and GuC (scheduling microcontroller) load or crash messages. Compute contexts die on reset; repeated hangs indicate firmware/kernel pairing problems more often than silicon.

• IOMMU DMAR Fault Causing GPU Hangs

  When GPUs attempt P2P transfers over the PCIe bus through the host's IOMMU, incorrect page table mapping or hardware bug in the CPU PCIe root complex causes a DMA access violation. The IOMMU blocks the transaction (DMAR fault), hanging the GPU's memory controller.

• KV Cache Corruption on Non-P2P Topologies

  KV cache corruption or NaNs in key_value tensors occurring during decoding on hardware topologies that do not fully support PCIe P2P memory sharing.

• Mellanox OFED Driver Mismatch

  New InfiniBand adapters (like ConnectX-7) require specific OFED versions. Mismatches with the host kernel or existing driver stack break the network fabric, making nodes unreachable or failing RDMA.

• MoE Expert Parallelism Comm Overlap Deadlock

  The MoE overlap scheduler uses a shared CUDA event as a 'baton' to sequence communications across nodes. Due to timing variations, ranks can launch overlapping NCCL collectives in different orders (e.g., Rank 0 launches All2All_A then All2All_B, while Rank 1 launches All2All_B then All2All_A). NCCL mandates that all ranks participating in a collective must submit the collective calls in the exact same sequence. The mismatch causes a cross-rank NCCL deadlock.

• NCCL P2P Level and Topology Routing Limits

  Improper NCCL P2P (Peer-to-Peer) configuration and topology routing lead to PCIe bandwidth bottlenecks, silently degrading multi-GPU training performance.

• NeuronCompilerError graph compilation timed out HloModule optimization Trainium

  The Neuron compiler exceeded its time budget compiling the graph. Trainium compiles the whole model graph ahead of execution, and compile time grows sharply with graph size and with the number of distinct shapes. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent custom-silicon failures.

• nrt_enqueue_dma failed device ring buffer full Neuron DMA channel saturated

  The Neuron runtime's DMA ring buffer filled, transfers were queued faster than the device drained them. The accelerator is keeping up with compute but not with the data being pushed at it. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent custom-silicon failures.

• NVIDIA NVML Initialization Failure

  The NVIDIA device plugin relies on the NVML library to discover GPUs and report their health to the kubelet. If the underlying GPU hardware experiences a fault (e.g., an Xid error causing it to fall off the PCIe bus), or if there is a cgroup driver mismatch (e.g., systemd vs cgroupfs) breaking container runtime permissions, NVML fails to initialize.

• NVIDIA Xid 119: GSP RPC Timeout Evidence and Recovery

  Preserve Xid 119 GSP timeout, driver and preceding event evidence. Coordinate recovery and verify the workload before concluding hardware failure.

• NVIDIA Xid 48. ECC Uncorrectable Memory Error

  Xid 48 is NVIDIA's code for uncorrectable GPU memory errors. It crashes training and corrupts model weights. Denpex correlates Xid 48 events with NCCL failures to identify the originating GPU and trigger automated draining.

• NVIDIA Xid 61 - Internal Microcontroller Breakpoint

  The GPU's internal processors (like the Power Management Unit or GSP) encountered a diagnostic event, such as a thermal anomaly, voltage droop, or firmware bug, and halted execution to prevent corruption.

• NVIDIA Xid 69 - Graphics Engine Class Error

  The GPU engine received an illegal command or entered an invalid state, commonly triggered by thermal throttling, power instability, or severe CUDA context corruption from conflicting workloads.

• NVIDIA Xid 74 - NVLink Fatal Error

  A physical hardware fault on the NVLink connection between GPUs. This can be caused by signal integrity degradation, mechanical stress, or connector contamination.

• NVIDIA Xid 79 - GPU Fallen off the Bus

  The CPU lost communication with the GPU over the PCIe bus. Common causes include inadequate power supply under high load, a loose PCIe connection, faulty riser cables, or active state power management (ASPM) conflicts.

• NVIDIA Xid 79. GPU Has Fallen Off the Bus

  Xid 79 means the GPU lost PCIe connectivity entirely. Training dies immediately. Denpex identifies which GPU, node, and the chain of events that led to the GPU fall. Root caused by bad PCIe, power events, or driver timeouts.

• NVIDIA Xid 94 - Contained ECC Memory Error

  The GPU hardware detected an uncorrectable memory error in VRAM. However, it successfully 'contained' the error to the specific application using that memory page, preventing a full system crash.

• nvlddmkm TDR (Timeout Detection and Recovery)

  The display driver crashes and recovers (or fails to recover) due to a long-running CUDA kernel triggering the OS watchdog.

• NVLink 5 switch tray ASIC port link training failed signal integrity

  An NVLink switch tray port failed link training, the physical layer could not establish a clean signal. On an NVL72-class rack this removes a path from the fabric, and collectives crossing it either fail or silently take a much slower route. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent blackwell failures.

• NVLink Error

  NVLink errors cause intra-node GPU communication failures and degraded multi-GPU performance.

• NVLink Fatal Error Xid 74 with Fabric Manager Crash

  A physical NVLink transceiver issue or NVSwitch port degradation causes high error rates. The Fabric Manager attempts to isolate the port but fails due to aggressive NCCL polling, resulting in a kernel panic or an unrecoverable Xid 74 error that drops the NVLink fabric.

• NVLink Link Down: Peer Evidence and Recovery Checks

  Distinguish an inactive topology link, NVLink errors and a failed peer. Preserve link identity and workload evidence before approved recovery.

• NVLink-C2C fatal link error cache coherency check failed Grace CPU Hopper GPU

  The coherent NVLink-C2C path between the Grace CPU and Hopper GPU reported a fatal link or coherency failure. Continuing on the node risks invalid data movement, so this is a platform health event rather than an application timeout. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent blackwell failures.

• NVLink/NVLS Failure with Xid 31 MMU Fault Cascade

  NVLS collective failures on Hopper/NVLINK4 nodes cascade from Fabric Manager restarts, producing Xid 31 MMU page faults and illegal memory accesses across the node.

• NVLS Illegal Memory Access (CUDA Error 700)

  NVLS (NVLink SHARP) offloads collective operations to the NVSwitch hardware. An incompatibility between the NCCL library version, the CUDA toolkit, and the NVIDIA display driver can cause incorrect memory mappings to be passed to the NVSwitch. When the switch attempts to read/write to the GPU memory space via these invalid addresses, it triggers a hard fault (Illegal Address).

• NVRM: Xid 31, MMU Fault: ENGINE GRAPHICS GPCCLIENT FAULT_PDE

  GPU MMU page fault (Xid 31), often an NVLS/Fabric Manager cascade. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent xid failures.

• NVRM: Xid 45, Preemptive cleanup

  Xid 45 records driver cleanup after a CUDA context was terminated. It is usually evidence of an earlier application exit, GPU reset, or administrator action, not the initiating hardware fault. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent xid failures.

• NVSwitch SXid errors: what aborts your CUDA jobs and what is only informational

  NVSwitch reports its own error family as SXid, not Xid. Whether an SXid aborts running jobs depends on severity and on which port reported it, and that distinction decides whether you reset one GPU or the whole baseboard pair.

• Partial NVLink Failure vs NVSwitch Failure (Localization)

  NVLink bandwidth degradation can be a single failing link or a whole-node NVSwitch fault. Comparing per-GPU NVLink bandwidth across all 8 GPUs tells them apart.

• PCIe ACS Blocking GPU Direct P2P

  PCIe Access Control Services (ACS) is enabled in the host motherboard's BIOS or the Linux kernel. ACS forces all PCIe peer-to-peer traffic to be routed up to the Root Complex (CPU) for IOMMU translation/security checks. NCCL detects this topology and disables direct GPU-to-GPU memory transfers (P2P) because routing through the root complex destroys performance and can cause hardware hangs.

• PCIe AER Uncorrectable Error Dropping GPU

  Marginal signal integrity on the PCIe riser cable or motherboard slot under heavy thermal load causes a Malformed Transaction Layer Packet (TLP). The PCIe root complex detects the fatal error via AER and isolates/resets the device, causing it to fall off the bus.

• PCIe Link Width 1x Max 16x PCIe Link Gen 1 Max 5 degraded

  PCIe link speed or width downgraded, the GPU is not running at Gen5 x16 (or expected width). This entry explains how to confirm the cause, apply the fix, and separate it from adjacent telemetry failures.

• PDN Resonance-Induced Silent Data Corruption

  Highly synchronous computational workloads (like dense GEMMs in LLMs) cause periodic power oscillations that resonate with the physical PDN, causing voltage droops that violate logic gate timing margins.

• PyTorch Pin_Memory Thread Deadlock

  When `pin_memory=True`, PyTorch spawns a background thread to copy pageable host memory to pinned (page-locked) host memory. If the objects returned by the Dataset are not standard tensors (e.g. custom objects, lists of un-pinnable types) or if there's a CUDA driver crash/OOM during the pinning process, the thread dies silently or gets stuck, and the main process hangs forever waiting for the pinned batch.

• RCCL WARN Ring failed IPC handle mapping failed across AMD Infinity Fabric gfx90a

  RCCL could not map a peer IPC handle across AMD Infinity Fabric, so the ring could not be formed. The AMD equivalent of an NCCL P2P transport failure, peer access between GPUs is unavailable or blocked. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent non-nvidia failures.

• Silent OOM Leading to Cluster-wide NCCL Timeout

  One specific worker experiences a CUDA OOM (e.g., due to memory fragmentation or an unusually large sample) and crashes. PyTorch's default process group error handling doesn't forcefully abort the remaining active ranks across different nodes. The surviving ranks hit the next dist.all_reduce barrier and wait forever for the dead rank.

• Silent PCIe Link Speed Downgrade Under Load

  Thermal expansion or microscopic dust in the PCIe slot causes transient connection issues during Active State Power Management (ASPM) or link retraining. The PCIe controller automatically negotiates a lower speed (Gen3) or narrower width (x8) to maintain stability.

• TopologyAffinityError Admission Rejection

  The Kubernetes scheduler is topology-blind. It assigns the Pod to a node based on total available capacity. However, the Kubelet's Topology Manager on the node (often configured with the 'single-numa-node' policy for high-performance GPU workloads) rejects the Pod at admission time because the requested CPU, memory, and GPU resources cannot be satisfied within a single NUMA domain.

• TPU Inter-Chip Interconnect ICI link error mesh coordinate link state DOWN

  A TPU ICI link at the reported mesh coordinate is down, so the accelerator slice cannot provide the topology expected by the distributed program. This is a slice or platform health fault, not an XLA graph error. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent custom-silicon failures.

• Uncorrectable Double Bit ECC Error

  Physical silicon degradation in the High Bandwidth Memory (HBM) modules causing multi-bit data corruption in a single memory word.

• Xid 149 NETIR_LINK_DOWN on Blackwell: the NVLink fault that Xid 74 knowledge misses

  On B100, B200 and GB200 class systems NVIDIA reports NVLink faults as Xid 149 with a NETIR subcode instead of the Xid 74 used on earlier generations. An operator or a runbook that only greps for Xid 74 sees nothing while the link is down and the GPU has already been pulled out of the NVLink domain.

• Xid 43: PCIe Link Training Failure Under Load

  Poor physical seating of the GPU in the PCIe slot, or a degraded PCIe riser cable, degrades signal integrity. When the system shifts into high-bandwidth PCIe Gen4/Gen5 states for inter-GPU communication, link training fails or bit errors overwhelm the error-correction. The resulting communication stall triggers the driver watchdog (Xid 43).

• Xid 43: Watchdog Timeout due to PSU Transient Spikes

  During heavy compute operations, modern GPUs (like the A100 or H100) experience massive microsecond-level power spikes. If the Power Supply Unit (PSU) or the 12V PCIe power cables cannot handle the transient load, the GPU voltage drops. This physical brownout stalls the GPU clocks, tripping the internal driver watchdog timer, which reports an Xid 43 (stopped processing) and resets the channel.

• Xid 48 Double-Bit ECC Error in HBM

  A multi-bit memory error occurred in the High Bandwidth Memory (HBM) that the hardware ECC engine cannot correct. To prevent silent data corruption, the GPU driver marks this as a 'sticky' hardware error and halts all compute operations on the device.

• Xid 48: Double Bit ECC Error from VRAM Degradation

  The physical VRAM (HBM or GDDR) on the GPU has degraded over time due to thermal stress or manufacturing defects. A double-bit error is mathematically uncorrectable by the GPU ECC engine, meaning data corruption has physically occurred in the memory cell. The driver kills the context to prevent silent data corruption.

• Xid 61 (Internal Micro-Controller Breakpoint)

  Often occurs randomly when the GPU utilizes aggressive power saving and attempts a rapid transition from its lowest state (PCIe Gen 1) to a high-performance state (Gen 3/4).

• Xid 62 (Internal Micro-Controller Halt)

  Caused by aggressive CPU contention (like massive dataloader thread counts) starving the NVIDIA kernel module of the CPU cycles required to acknowledge GPU interrupts.

• Xid 62: Internal microcontroller halt

  Xid 62 reports an internal microcontroller halt. It does not by itself distinguish a driver or firmware defect from faulty hardware, and does not establish a CUDA memory leak.

• Xid 63 Page Retirement Storm causing severe throttling

  The HBM is rapidly degrading, triggering single-bit ECC errors at an extremely high frequency. The GPU's self-healing mechanism dynamically retires and remaps these bad memory pages (Xid 63). Doing this hundreds of times per second stalls the memory controller and destroys memory bandwidth.

• Xid 64 InfoROM Page Retirement Failure

  The GPU detected a bad HBM memory page and attempted to record it in the InfoROM (EEPROM) for permanent retirement. However, the InfoROM is either full or physically degraded, preventing the page remap table from being updated. The driver halts initialization to prevent data corruption.

• Xid 79: GPU has fallen off the bus

  Xid 79 means the NVIDIA driver cannot reach the GPU over PCIe. Investigate link, device, and driver evidence before deciding on recovery or replacement.

• Xid 92: High single-bit ECC error rate

  Xid 92 reports a high rate of corrected single-bit ECC errors. Corrected errors preserve the original data; this event does not prove uncorrectable memory errors, corrupted tensors, or the cause of NaN loss.