CORA slashes fine-tuning parameters 4x over LoRA with SVD rotation

Parameter-efficient fine-tuning (PEFT) has become essential for adapting large models, but existing low-rank approaches like LoRA fail to preserve the coupled geometry between pretrained singular bases. In a new paper, Wang et al. introduce CORA (Coherent Orthogonal Rotation Adaptation), which builds on minimum-perturbation theory to apply a single orthogonal rotation simultaneously to both left and right singular bases per slice of the weight matrix. This per-slice approach allows the model to adapt in a geometrically coherent way, requiring only ½m(r−1) trainable parameters per linear layer—roughly 4× fewer than standard LoRA at the same rank.

Empirically, CORA outperforms LoRA, DoRA, PiSSA, and MiLoRA across commonsense reasoning and code generation tasks while using up to 8× fewer total parameters. The method works by decomposing the pretrained weight via rank-r SVD, then applying per-slice orthogonal rotations and a per-layer diagonal scaling. This results in a highly efficient adaptation that is both theoretically grounded and practically superior. For practitioners, CORA offers a drop-in replacement for LoRA that reduces memory and compute overhead without sacrificing—and often improving—model quality. The paper is available on arXiv under 2607.02576.