Split-FG slashes training memory 35% with backprop-free method

Backpropagation's memory footprint is a major bottleneck for training large models, as it must store intermediate activations for the backward pass. Forward-mode gradient methods avoid this cost but suffer from noisy estimates that worsen as parameter count grows. Split-FG, proposed by Tian Qin and Wei-Min Huang, splits the network at an intermediate representation: it computes the output head gradient exactly (small number of parameters) and estimates only the trunk gradient using a Jacobian–vector product, requiring no backward pass through the trunk. This hybrid approach dramatically reduces estimator variance while retaining Adam-style convergence guarantees. The authors also identify a critical failure mode: naive forward-gradient trunk training performs worse than leaving the trunk randomly frozen, because Adam applies overly aggressive updates to noisy, under-determined trunk coordinates. Simply using a much smaller learning rate for the trunk reverses this outcome.

On WikiText-103, a 16M-parameter GPT-2-style model trained with Split-FG reaches a validation perplexity of 387, compared to 668 for a frozen-trunk control and 2,885 for a pure forward-gradient baseline (backpropagation achieves 150). Split-FG also produces the strongest backprop-free results on tabular benchmarks and reaches 60.5% accuracy on CIFAR-10 and 35.2% on CIFAR-100 with a heavy-head design. Peak memory is reduced by up to 35% relative to matched backpropagation, though the performance gap widens as the forward-mode trunk grows. This work offers a practical path to training large networks on memory-constrained hardware, potentially democratizing deep learning for teams without access to high-end GPUs.