Developer Tools

PyTorch's new CUPTI callback registry enables CUDA graph node annotation

PR #191944 lets multiple profiler consumers share one CUPTI subscription without fatal conflicts.

Deep Dive

PyTorch PR #191944 adds a shared CUPTI subscriber-callback registry to the profiler monitor, so consumers can consume CUPTI subscriber callbacks without holding a second subscription. Previously the monitor's subscription only handled activity records, and its subscriber callback was a no-op. The motivating consumer is CUDA-graph node annotation via GRAPHNODE_CREATED, but that consumer is not part of this change — only the mechanism is. The registry avoids Kineto's MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED failure by sharing the monitor's subscription, and it uses refcounted arming per (domain, cbid) rather than one shared enable flag. Dispatch runs on the application thread without a lock, swaps in fresh immutable tuples instead of mutating in place, and swallows per-handler exceptions so one bad handler can't silence the rest. Tests: 8 registry tests passed, including a smoke test arming GRAPHNODE_CREATED over a real CUDA-graph capture; 60 regression tests passed with 2 pre-existing skips; and 7 node timer tests passed. The lone warning in the suite is pre-existing and not introduced by this change.

Key Points
  • Adds a shared CUPTI subscriber-callback registry keyed by (domain, cbid) in the PyTorch profiler monitor
  • Prevents MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED crashes by eliminating the need for a second CUPTI subscription
  • Enables CUDA graph node annotation via GRAPHNODE_CREATED, with 75 tests passing across registry and regression suites

Why It Matters

Unlocks CUDA-graph-level profiling in PyTorch while keeping the profiler stable for existing GPU workloads.

📬 Get the top 10 AI stories daily