BaLoRA: Bayesian LoRA boosts accuracy and uncertainty for fine-tuning

Low-Rank Adaptation (LoRA) has become the standard for fine-tuning large pre-trained models while keeping computational costs low. However, its low-rank point-estimate updates limit expressiveness, leaving a persistent accuracy gap compared to full fine-tuning and offering no built-in uncertainty quantification. This makes LoRA less suitable for high-stakes applications where reliability matters as much as accuracy. To address these limitations, Dario Coscia, Sindy Löwe, and Max Welling introduce BaLoRA (Bayesian Low-Rank Adaptation), a novel extension that incorporates an input-adaptive Bayesian parameterization of LoRA matrices. This adds only a minimal number of parameters and computational overhead while providing well-calibrated uncertainty estimates. Surprisingly, the adaptive noise injection inherent to the Bayesian approach also significantly improves prediction accuracy, narrowing the performance gap with full fine-tuning across both natural language reasoning and vision tasks.

In practical evaluations, BaLoRA demonstrates strong results on a real-world scientific task: band gap prediction in metal-organic frameworks. The method produces zero-shot test-time uncertainty estimates that correlate more strongly with model error than those from a trained ensemble of standard LoRA models. Moreover, these uncertainty estimates improve monotonically with computational budget without sacrificing accuracy. This combination of improved accuracy and reliable uncertainty quantification makes BaLoRA a promising advancement for fine-tuning large models in domains where trust and precision are critical, such as scientific discovery, healthcare, and autonomous systems. The work is published on arXiv as paper 2605.08110.