Developer Tools

PyTorch's CUPTI backend speeds CUDA-graph annotation 2x for nested scopes

New CUPTI-based node attribution runs 2.2x faster and fixes missing annotations

Deep Dive

PyTorch's latest commit introduces a second backend for annotating CUDA-graph nodes, using CUPTI node-creation callbacks instead of the existing dependent-edge walk. The new approach, controlled by torch.cuda.graph(annotation_config={"backend": ...}), registers a RESOURCE/GRAPHNODE_CREATED handler that records each node against the open scope when CUPTI announces its creation. The "auto" setting uses CUPTI when a monitor subscription exists, otherwise defaults to the original walk. This fixes a correctness gap where scopes entered while the current stream wasn't yet capturing were completely missed—measuring 0 annotated nodes versus 3 with CUPTI.

Performance also improves dramatically. The old walk rescans nested scopes once per enclosing scope, making cost quadratic in nesting depth. With CUPTI, it's linear: at depth 16 over roughly 2,550 nodes, annotation overhead drops from 73.6ms to 33.5ms (isolated from the 18.2ms capture cost). Additionally, the change consolidates capture-graph-id reads, removing a cudaGraphGetId call per capture. Note that the CUPTI backend requires single-threaded autograd; if that gate fails, "auto" falls back to the walk, while explicit "cupti" raises an error. Nodes from child graphs or conditional bodies are dropped rather than recorded.

Key Points
  • New CUPTI callback backend for mark_kernels fixes 0-node annotation bug on non-capturing streams
  • Annotation overhead drops from 73.6ms to 33.5ms at depth 16 (~2550 nodes), making cost linear vs quadratic
  • Config via torch.cuda.graph(annotation_config={"backend": "auto"}) with typo validation on keys and values

Why It Matters

Profiling CUDA graphs becomes faster and more accurate, enabling deeper introspection for PyTorch performance engineers.

📬 Get the top 10 AI stories daily