What does a dcgmi diag -r 3 failure indicate?

A Level 3 DCGM diagnostic failure indicates rigorous stress testing has uncovered hardware degradation across GPU memory HBM, PCIe bus lanes, NVLink connections, or compute tensor cores.

How do PCIe AER Corrected errors differ from Uncorrected Fatal bus errors?

PCIe Corrected errors are transparently handled by hardware retry mechanisms without dropping training jobs. Uncorrected Fatal errors cause bus lockups, Xid 79 faults, and require physical hardware replacement.

When is a GPU memory or NVLink failure eligible for manufacturer RMA?

Uncorrectable ECC memory errors, persistent Xid 79 fallen off the bus events, and permanent physical link training failures qualify the GPU for vendor RMA replacement.

How can PCIe Active State Power Management (ASPM) cause GPU link dropouts?

Aggressive PCIe ASPM power states can cause link power down transitions during heavy deep learning data transfers, resulting in replay timer timeouts and bus resets. Disabling ASPM in kernel boot parameters prevents this.

What is the difference between DCGM Level 1, Level 2, and Level 3 diagnostic suites?

Level 1 runs quick hardware availability checks. Level 2 performs comprehensive memory and interconnect stress tests. Level 3 executes intensive multi-subsystem fabric and hardware validation under full load.

DCGM & PCIe AER Diagnostic Error Decoder

Paste raw dcgmi diagnostic reports, kernel dmesg PCIe AER logs, or NVIDIA DCGM health monitor outputs. Instantly isolate hardware faults, triage GPU link degradations, and generate remediation scripts.

Raw Diagnostic Log or Kernel TraceFormat: DCGMI_DIAG

Quick Sample Presets

Cluster Health Summary

Overall StatusCritical Failure

Affected GPUs0

RMA Candidates1 GPU

Findings Count5

Triage Findings & Actionable Fixes (5)

NVLink / NVSwitchGPU 0

Warning Degraded

Test / Error Code: NVVS

| NVVS : Fail |

Root Cause: NVLink high error rate or link training failure between NVSwitch and GPU.

Actionable Fix: Query NVLink status and check physical bridge connectors or NVSwitch health.

$ dcgmi nvlink -e

GPU Memory (HBM/GDDR)GPU 0

RMA RequiredCritical RMA

Test / Error Code: Memory

| Memory : Fail |

Root Cause: Uncorrectable ECC memory failure or failing memory controller cells.

Actionable Fix: Memory subsystem has unrecoverable errors. RMA recommended.

$ nvidia-smi -q -d MEMORY

PCIe BusGPU 0

Warning Degraded

Test / Error Code: PCIe

| PCIe : Fail |

Root Cause: PCIe lane degradation, signal integrity loss, or slot seating issue.

Actionable Fix: Reseat the GPU in its PCIe slot, verify power cables, and check motherboard BIOS settings.

$ lspci -vvv -s 0000:03:00.0

NVLink / NVSwitchGPU 0

Warning Degraded

Test / Error Code: NVLink

| NVLink : Fail |

Root Cause: NVLink high error rate or link training failure between NVSwitch and GPU.

Actionable Fix: Query NVLink status and check physical bridge connectors or NVSwitch health.

$ dcgmi nvlink -e

Compute Core (SM)GPU 0

Warning Degraded

Test / Error Code: Status

| Status: Failed (Errors found during diagnostic execution) |

Root Cause: Diagnostic test failed during execution suite.

Actionable Fix: Reseat hardware or run targeted diagnostics.

$ dcgmi diag -r 2

Automate GPU Cluster Diagnostics with Denpex

Deploy Denpex machine learning infrastructure reliability agents to automatically diagnose PyTorch training failures, NCCL timeouts, and GPU hardware faults across Kubernetes clusters.

Start Free Team Evaluation