PyTorch adds CUPTI PM-sampling engine for zero-overhead GPU profiling
New engine tracks SM, DRAM, NVLink metrics with negligible GPU cost.
Deep Dive
The PyTorch team introduced a new CUPTI-based PM-sampling engine (PmSampler) for continuous, low-overhead monitoring of on-chip GPU performance counters, such as SM activity, DRAM throughput, and NVLink bandwidth. The engine is a per-device singleton, opt‑in (as it locks GPU clocks, affecting kernel timing), and uses a pull‑based architecture where multiple consumers each poll the same session for their own metrics.
Key Points
- Single-pass collection of union metrics from all consumers, with rejection on unsupported combinations.
- Per-device singleton with refcounting to safely manage multiple collectors and teardown without segfaults.
- Pull-based polling with HW timestamps in CUPTI clock domain; ring buffer sized by TORCH_CUPTI_PM_SAMPLING_INTERVAL_MS and LOOKBACK_MS.
Why It Matters
Enables continuous, low-overhead GPU performance monitoring for AI workloads, helping developers optimize model training and inference.