arXiv's FPO fine-tunes LLMs 3.2x faster without backpropagation

A new arXiv paper from Rivaan Patil, Simon Dennis, Hao Guo, and Kevin Shabahang introduces Forward-Pass-Only MLP training (FPO), a method that adapts large language models without a backward pass through the model body. FPO achieves 2.7–3.2x the throughput of standard fine-tuning while using roughly 40% less peak training memory, and it leaves off-domain benchmarks like MMLU, ARC-Challenge, HellaSwag, and Winogrande within seed-noise of baseline—a property full-network fine-tuning doesn't reliably reproduce. The key trick: FPO computes a single error signal at the output and applies it directly to each target layer, bypassing cross-layer backpropagation entirely and avoiding autograd graph construction at any point.

The method rests on an empirical observation: at late transformer layers, the output-layer prediction error approximates the true gradient with cosine similarity 0.47–0.59 across six public models. The authors also provide a two-minute diagnostic to quantify this approximation per layer, identifying where late-layer adaptation is viable. FPO was evaluated on three model families—OLMo-2-7B, Qwen3-8B, and Falcon3-7B—and produced in-domain perplexity improvements in all cases. Localizing standard SFT to FPO's target layers can also enter this regime, but at 2.2x the wall-clock cost of FPO. For practitioners, this suggests a cheaper, faster path to domain adaptation that preserves general knowledge, potentially enabling fine-tuning on more modest hardware.