PyTorch AD from Scratch: 22 Gradients in PINNs Traced Step by Step
A new paper manually traces every gradient PyTorch computes for PINNs — with 22 parameters.
Most developers trust PyTorch's autograd as a black box, but few understand exactly how gradients flow through a computational graph — especially in the two-tiered world of Physics-Informed Neural Networks (PINNs). A new educational paper by Abdeladhim Tahimi walks through every single node and adjoint value, using a toy 1-3-3-1 multilayer perceptron and the initial value problem y'(t)+y(t)=0, y(0)=1. The result is a rare, numerically verified trace of PyTorch's reverse-mode AD at work.
Tahimi's tutorial covers three key mechanisms: building the forward-pass computational graph, running reverse-mode backward traversal to collect 22 parameter gradients in a single pass, and — critically — enabling create_graph=True so that the physics residual itself can be differentiated. Every adjoint value is cross-checked against the P/Q sensitivity framework from the author's earlier work (2026). The paper is dense with 9 figures, 25 pages, and explicit numerical values at every step, making it an essential resource for ML practitioners who want to move past API-level understanding of automatic differentiation.
- Traces reverse-mode AD for a 1-3-3-1 MLP with 22 parameters, verifying every gradient against hand derivations.
- Explains create_graph=True mechanism needed for PINNs, where the loss depends on the derivative of the network output.
- Uses a concrete ODE (y'(t)+y(t)=0) to illustrate how physics-informed residuals propagate through autograd.
Why It Matters
Demystifies PyTorch's autograd for PINN training, enabling better debugging and custom gradient workflows.