FuRA: Spectral preconditioning beats LoRA with full-rank updates
LoRA’s low-rank bottleneck isn’t inevitable—FuRA proves you can have full-rank updates with near-identical memory and speed, rewriting the efficiency trade-off that has dominated LLM adaptation.
Traditional fine-tuning methods like full fine-tuning and LoRA ignore the spectral structure established during pretraining, allowing noisy gradients from limited data to perturb robust features. FuRA (Full-Rank Adaptation) introduces spectral preconditioning by reparameterizing each weight matrix through its full-rank singular value decomposition (SVD) and freezing one singular basis. This is implemented via a block tensor-train factorization W = LSR, where the large core L is fixed to the pretrained block-wise SVD basis, while only the compact core R and block-wise singular values S are optimized. This design provides full-rank spectral preconditioning, preserves full-rank update expressivity, and achieves parameter, memory, and step-time efficiency comparable to LoRA.
On benchmarks, FuRA consistently outperforms full fine-tuning across multiple settings: +1.37 on LLaMA-3-8B commonsense reasoning, strong results in LLM reinforcement learning for mathematical reasoning, and visual instruction tuning for VLMs. The 4-bit quantized variant QFuRA also surpasses QLoRA, demonstrating practical deployment advantages. By combining full-rank expressivity with parameter-efficient memory usage, FuRA sets a new state-of-the-art for fine-tuning large models, offering a drop-in replacement for LoRA and full FT without extra compute cost.
- FuRA achieves full-rank fine-tuning with LoRA-level parameter counts by using spectral preconditioning via block tensor-train factorization, outperforming full fine-tuning on LLaMA-3-8B.
- The 4-bit variant QFuRA surpasses QLoRA, suggesting that quantization-aware methods can benefit from preserving full-rank updates.
- The core insight—that update subspaces can be compactly represented without rank truncation—may reshape PEFT tools, but its dependence on a fixed SVD basis introduces risks for out-of-distribution tasks.
Why It Matters
FuRA proves full-rank fine-tuning can be efficient, challenging the low-rank dogma that has defined PEFT since LoRA.