PyTorch optimizes autograd function with ArrayRef, cuts instructions 5.7%
Commit avoids refcount bumps in autograd.Function.apply, improving performance for all users.
PyTorch’s autograd engine just got a performance boost thanks to a focused optimization in `autograd.Function.apply`. Contributor zou3519 merged commit #189788, which changes how the function stores Python tensor inputs during execution. Instead of creating an owned `variable_list` (which requires bumping refcounts on every `TensorImpl`), the new code uses a `SmallVector<const Variable*, 24>` that is compatible with `ArrayRef`. This means the common path now borrows tensor handles, avoiding unnecessary reference counting overhead. The change is carefully scoped: tracing hooks and the user-defined JVP (Jacobian-vector product) callback still receive an owned `variable_list` when they need to hold onto the tensors beyond the apply call.
The performance impact is tangible. On a local benchmark that exercises a single `apply` call with 13 inputs and 2 outputs (and no saved tensors), the instruction count dropped from 54,090 to 51,030 — a reduction of about 5.7%. While that might seem small per call, `autograd.Function.apply` is used extensively in custom layers, distributed training, and custom gradient pathways. The improvement compounds across thousands or millions of forward/backward passes. The commit was approved by soulitzer and depends on two prior PRs (#189577 and #189582) that likely set up the `ArrayRef` infrastructure. This is a textbook example of a low-level memory management tweak that benefits every PyTorch user without changing the API.
- Replaces owned `variable_list` with `ArrayRef<const Variable*>` on common autograd apply path
- Instruction count reduced from 54,090 to 51,030 in benchmark (5.7% improvement)
- Maintains owned variable_list for tracing and user JVP callbacks where needed
Why It Matters
A 5.7% reduction in autograd overhead speeds up custom layers and distributed training for all PyTorch users.