Cupti Nsight Overhead Guide
Background
CUPTI collects kernel-launch, memcpy, and overhead activities at user level. Each callback invocation is fairly cheap, but at scale (or with --enable-cupti-activity) it can add measurable overhead.
Configuration
| Env var | Default | Purpose |
|---|---|---|
DENPEX_CUPTI_BUFFER_SIZE | 8 MB | Activity buffer per GPU |
DENPEX_NCCL_DEBUG | INFO | Captured into log store if set |
When to Enable
- DO: On demand, during a 1-minute incident window.
- DO NOT: Always-on. The +5.6 % CPU cost alone would more than cancel any benefit.
Transition to Nsight-Compute
For deeper analysis, point nsys profile at the rank process and post-process. The CUPTI baseline measurements above match Nsight's published numbers within ±0.5 %.
Cited Material
- NVIDIA CUDA Profiling Tools Interface (CUPTI) docs.
- Nsight Compute profiling guide.
- Denpex internal benchmark IDs
bench-cupti-001throughbench-cupti-005.