Developer Tools

PyTorch adds shared CUPTI callback registry for CUDA graph annotation

New PR fixes MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED to enable graph node profiling

Deep Dive

PyTorch's profiler now supports consuming CUPTI subscriber callbacks through a shared registry, addressing a long-standing limitation. Previously, the CUPTI monitor's subscription was shared only for activity records; subscriber callbacks (which fire synchronously on the application thread during CUDA API calls) had no path through the monitor. The motivating use case is CUDA graph node annotation, which requires the GRAPHNODE_CREATED callback to learn which nodes a capture scope contains. Users couldn't register their own subscriber: holding a second cuptiSubscribe_v2 across Kineto's first profile causes initCallbackApi to fail with MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED, and Kineto never retries. This new registry in the monitor solves that by sharing the monitor's existing subscription.

The implementation is careful about performance and safety. Callback registration is separate from arming, which is refcounted per (domain, cbid) so multiple consumers can scope delivery without disabling each other. Dispatch runs on the application thread inside the CUDA call, so it deliberately avoids locks by swapping in immutable tuples rather than mutating state. It also swallows per-handler exceptions to prevent one bad handler from silencing the rest. Test coverage includes a smoke test that arms GRAPHNODE_CREATED over a real CUDA-graph capture and asserts the invocation count freezes across replays once disarmed. In total, 67 tests pass, confirming the activity-path regression split between register_callbacks() and stop() works cleanly. This change only adds the mechanism; the actual CUDA-graph annotation consumer will ship in a future PR.

Key Points
  • Adds a shared CUPTI subscriber-callback registry to PyTorch's monitor (PR #191944)
  • Avoids MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED by letting consumers share the monitor's subscription instead of creating a second one
  • Refcounted arming per (domain, cbid) and lock-free dispatch; 67 tests pass including CUDA graph capture smoke tests

Why It Matters

Enables reliable CUDA graph node annotation in PyTorch profiler without breaking Kineto's existing subscription.

📬 Get the top 10 AI stories daily