PyTorch speeds up CUDA graphs 2.65x by skipping empty kernel captures
Empty CUDA graph warning eliminated, with up to 2.65x faster steady-state execution.
PyTorch's latest optimization (PR #185760) eliminates unnecessary CUDA graph captures for kernel-free inductor graphs. The issue arose when operations like view-only reshapes were lowered to CUDA graphs that had no kernel launches, triggering an empty CUDA Graph warning and adding overhead. Author jansel implemented a fix that inspects the lowered scheduler after codegen: if only NopKernelSchedulerNode (no compute kernels) remains, cudagraphs are disabled. This is more reliable than FX-level guessing, as it uses actual lowered work rather than op semantics.
Benchmarks confirm big gains: median first 3 calls fell from 0.052s to 0.046s, and median steady-state over 5000 calls plummeted from 0.243s to 0.091s — over 2.65x faster. The fix also cleans up counters: before it incremented 'cudagraph_recorded_non_static_inputs', now it correctly uses 'cudagraph_skips'. For developers using CUDA graphs with PyTorch Inductor, this means reduced overhead and cleaner warnings for view-heavy workloads, while compute-heavy graphs still use cudagraphs as expected.
- Fix skips CUDA graph capture for graphs with only NopKernelSchedulerNode (e.g., view reshaping).
- Steady-state throughput improved 2.65x: from 0.243s to 0.091s over 5000 calls.
- Removes spurious 'empty CUDA Graph' warning and uses actual lowered work check instead of FX-level guessing.
Why It Matters
Developers using PyTorch CUDA graphs get faster view-heavy operations and cleaner debugging feedback.