CUDA Device Assertion Failure

CUDA device assertion failures indicate GPU hardware problems that crash training with ambiguous error messages. Denpex maps the device assertion to the specific GPU and operation, distinguishing hardware faults from driver issues.

CUDA Device Assertion Failure is a Hardware failure seen during ML training runs. CUDA device assertion failures indicate GPU hardware problems that crash training with ambiguous error messages. Denpex maps the device assertion to the specific GPU and operation, distinguishing hardware faults from driver issues. Common tags: Device Assert, Cuda, Hardware, Gpu.

Single-bit or double-bit ECC error in GPU memory corrupts tensor data during a kernel operation, causing an assertion in downstream operations (e.g., index out of bounds in embedding lookup). NVLink transaction failure corrupts data transferred between GPUs in the same node, detected by a device-side bounds check. GPU memory channel degradation causes intermittent bit flips that only manifest under specific memory access patterns. GPU clock or voltage instability from aging hardware causes computation errors in CUDA cores during sustained high utilization. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training crashes with 'CUDA error: device-side assert triggered' and no further context
• The error is non-reproducible: restarting on different GPUs may succeed
• The assertion provides no information about which kernel or tensor triggered it

• NVIDIA GPUs with hardware faults (ECC errors, NVLink issues, memory channel failures)
• Training with CUDA graphs that reuse cached kernel launches across faulty hardware
• Long-running training jobs that accumulate hardware errors over days or weeks
• High-utilization clusters where GPUs are not regularly stress-tested

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: RuntimeError: CUDA error: device-side assert triggered at /pytorch/aten/src/ATen/native/cuda/.
• ✓Verified signal present: The same training config crashes on some GPUs but not others in the same cluster
• ✓Verified signal present: nvidia-smi shows Xid errors (14, 48, 64) associated with the failing GPU
• ✓Verified signal present: CUDA kernel launches succeed but produce garbage output before triggering the assertion
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

CUDA reports errors asynchronously, so the traceback usually points at whatever line synchronised next rather than the one at fault. The hub covers every common CUDA error and how to make it report honestly.

• Single-bit or double-bit ECC error in GPU memory corrupts tensor data during a kernel operation, causing an assertion in downstream operations (e.g., index out of bounds in embedding lookup)
• NVLink transaction failure corrupts data transferred between GPUs in the same node, detected by a device-side bounds check
• GPU memory channel degradation causes intermittent bit flips that only manifest under specific memory access patterns
• GPU clock or voltage instability from aging hardware causes computation errors in CUDA cores during sustained high utilization