Gpu Profiling Overhead Study
When --enable-cupti is enabled, Denpex subscribes to CUPTI activity and callback APIs. This study quantifies the overhead.
Measurement Methodology
- 4-GPU node, single-rank workload.
- Workload: ResNet-50, batch size 64, AMP, eager mode.
- 5 trials, median reported.
Results
| Mode | Without CUPTI | With CUPTI Activity | With CUPTI Activity + Callback |
|---|---|---|---|
| CPU | 0.0 % | +3.1 % | +5.6 % |
| Memory | 0 MB | +220 MB | +360 MB |
Recommendation
CUPTI is intended for incident mode only — the operator turns it on for a single 60-second window when investigating, then off again. The published overhead numbers reflect this short-window behavior.
Falsification
If you reproduce these numbers and exceed the budget by more than ±5 %, file a bug. The reproduction script lives at benchmarks/cupti-overhead/run.sh.