Cuda Failure Guide

What Denpex Catches

Setup

The agent uses libnvidia-ml.so.1 (NVML) to monitor device state and libdcgm.so (DCGM) for field-level counters.

Common Symptoms and Causes

Symptoms: rank hangs at first AllReduce

• Probably the CUDA driver is mismatched. Check nvidia-smi --query-gpu=driver_version --format=csv,noheader.
• Expected: matches the runtime torch.version.cuda.

Symptoms: steady-state jobs see DBE

• Often a sign of a dying HBM page. Look for row_remap_pending in NVML or DCGM.
• DCGM Level-3 will confirm whether the device is recoverable.

Symptoms: torch.cuda.OutOfMemoryError at step 5k

• Could be a memory leak in user code or a fragmentation issue.
• Denpex captures the snapshot of allocations at the OOM time and emits TrainingEvent::OomMemorySnapshot.

Symptoms: Xid 79 (Fall off the bus)

• Hardware fault. Denpex drains the node, reroutes the job, issues RMA.

Limits

We do NOT capture:

• Full kernel PTX.
• Full instruction pointers.
• Full register snapshots.

We DO capture:

• Kernel launch parameters.
• ECC counters.
• Thermal counters.
• PCIe counters.

Reproductions

docs/benchmarks/cuda-failure-injection/ injects each Xid in a 4-GPU reduced testbed and asserts the rule fires within 60s.

See Also

• docs/agent-architecture.md — what the agent monitors.
• docs/hardware-degradation-validation.md — predictive detection.