AuroOFT fine-tuning beats QLoRA by 10.62% on quantized LLMs
New AuroOFT method adds nonlinear gating to QOFT, slashing trainable parameters by 44.7%.
Quantized orthogonal fine-tuning (QOFT) has enabled parameter-efficient adaptation of low-bit language models by learning structured activation rotations before frozen quantized weights. However, these updates are limited to linear orthogonal transformations, which restricts the model's ability to capture input-dependent nonlinear corrections. To address this, Yue Han and Dianlin Wang propose AuroOFT, which keeps QOFT as a stable quantization-compatible branch while attaching a zero-start gated low-rank nonlinear residual to each adapted linear layer.
AuroOFT maps activations into an RMS-normalized compact latent space and uses adaptive nonlinear bases with bounded or token-dependent gating. The zero-initialized up projection makes AuroOFT functionally identical to QOFT at initialization, while orthogonality remains a branch-level stability property. Under matched protocols, AuroOFT improves Macro-6 by 1.30-2.70% over QOFT on Qwen2.5 1.5B/3B settings, exceeds QLoRA by 6.52-10.62%, and saves 32.3-44.7% trainable parameters relative to QLoRA. The method's code is available via an anonymous repository, making it accessible for further research and adoption.
- AuroOFT combines QOFT with a zero-start gated low-rank nonlinear residual for expressive adaptation.
- Improves Macro-6 by 6.52-10.62% over QLoRA on Qwen2.5 1.5B/3B models.
- Uses 32.3-44.7% fewer trainable parameters than QLoRA, enabling efficient low-bit LLM fine-tuning.
Why It Matters
AuroOFT enables cheaper, more accurate fine-tuning of quantized LLMs, reducing memory and compute needs for deployment.