PyTorch Just Quietly Cut CUDA Memory by 16% for RAdam — Here’s the Dynamo Trick
A single-line fix slashes memory usage in PyTorch's JIT compiler for optimizer loops.
PyTorch's Dynamo, the just-in-time (JIT) compiler for accelerating PyTorch programs, was found to allocate excess memory when optimizing certain optimizers like RAdam and NAdam. The issue occurred because Dynamo decomposed `torch._foreach_lerp_` whenever it encountered a scalar overload, splitting the single operation into a `_foreach_sub`, per-tensor weight materialization, and `_foreach_addcmul_`. For Python scalar weights, this decomposition generated additional full-size temporary tensors that pure eager execution never creates, causing peak CUDA memory allocation to spike by about 16% for RAdam.
A new commit by jansel (#186452) fixes this by preserving the native `foreach_lerp_` call when the weights are Python scalars. Only tensor weights still trigger the decomposition to avoid graph breaks. The result: compiled code now matches eager memory usage exactly. For RAdam, peak memory went from [256000, 256000, 256000] bytes to [220160, 220160, 220160] bytes—identical to eager. NAdam similarly now matches eager at [266240, 266240, 266240]. The fix was tested via dedicated unit tests and confirmed with a peak-memory reproduction script from issue #125511.
- Dynamo's previous decomposition of scalar `foreach_lerp_` caused 16% higher peak CUDA memory (e.g., RAdam: 256MB vs 220MB).
- The fix retains the native `foreach_lerp_` call for Python scalar weights, matching eager allocation.
- Tensor weights still get decomposed, preserving graph-break prevention. Memory benchmarks now identical to eager for RAdam and NAdam.
Why It Matters
Reduces memory overhead for PyTorch compiled optimizers, enabling larger batch sizes or models without OOM.