DCGM vs nvidia-smi vs Compute Sanitizer Triage Guide
Diagnosing GPU faults requires choosing the right diagnostic tool for the symptom. NVIDIA provides nvidia-smi for instantaneous state checks, DCGM for continuous fleet telemetry and active validation, and Compute Sanitizer for kernel memory corruption. This guide compares their operational use cases.
Comparison Matrix
| Dimension | NVIDIA nvidia-smi | NVIDIA DCGM (Data Center GPU Manager) | NVIDIA Compute Sanitizer |
|---|---|---|---|
| Primary Operational Scope | Ad-hoc, instantaneous query of GPU clock speed, temperature, and ECC counts. | Continuous background telemetry daemon, health monitoring, and active stress diagnostics. | Kernel-level memory access validation, out-of-bounds checks, and race condition detection. |
| When to Run | Immediately after a job crash to inspect instantaneous error states and throttling flags. | Continuously in production clusters and before job launches via dcgmi diag -r 3. | During pre-production kernel development and when debugging SIGSEGV / illegal memory access. |
| Key Metrics & Output | Power draw, temperature, fan speed, active processes, and ECC error tallies. | Aggregated health status, PCIe bandwidth utilization, Xid counters, and reliability scores. | Detailed kernel line numbers, warp error addresses, shared memory out-of-bounds logs. |
| Performance Overhead | Negligible instantaneous CLI query overhead. | Low, configurable daemon footprint designed for 24/7 production monitoring. | High runtime performance overhead (runs kernels under a virtualized execution checker). |
| Integration & Automation | Easily wrapped in bash monitoring scripts and cron jobs. | Prometheus exporter integration, Kubernetes operator hooks, and REST APIs. | Integrated into unit test suites and CI/CD GPU validation pipelines. |
Signature failures and first triage commands
nvidia-smi Instantaneous Query
GPU 0: NVIDIA H100 SXM5 (UUID: GPU-...) | Temp: 42C Pwr: 75W / 700W | ECC: 0 Errorsnvidia-smi -q -d ECC,POWER,PERFORMANCEDCGM Diagnostic Health Check
DCGM Diagnostics Enterprise Health Check: PASS (Code 0) - All power, memory, and PCIe links verified.dcgmi diag -r 3Compute Sanitizer Out-of-Bounds Trap
========= Invalid __global__ read of size 4 bytes at 0x... in kernel vector_addcompute-sanitizer --tool memcheck python3 train.pyEngineering decision guide
Use nvidia-smi for Quick Triage & Spot Checks
Run nvidia-smi first when an engineer reports a frozen node or hanging job to instantly check if a GPU has dropped off the bus or hit a thermal ceiling.
Use DCGM for Fleet-Wide Telemetry & Automated Health Audits
Deploy DCGM across your entire cluster to continuously collect metrics into Prometheus and run rigorous pre-flight hardware diagnostics before starting multi-node jobs.
Use Compute Sanitizer for Deep Code-Level Kernel Debugging
Use Compute Sanitizer when encountering cryptic CUDA illegal memory access errors during custom CUDA kernel development or Triton compilation.
Frequently asked questions
Can nvidia-smi replace DCGM in a production cluster?
No. nvidia-smi provides point-in-time snapshots and has rate-limiting issues if polled too rapidly. DCGM is specifically engineered for low-overhead continuous telemetry and active diagnostic stress testing.
What does dcgmi diag level 3 test?
Level 3 is a comprehensive diagnostic suite that tests PCIe bandwidth, target memory stress, targeted CUDA core calculations, and NVLink fabric integrity under load.
Does Compute Sanitizer slow down training execution?
Significantly. Because Compute Sanitizer instruments every memory reference, execution speed drops by 10x to 100x. It should be used for debugging, never for continuous production training.