Rank-Accuracy Trade-off for LoRA: A Gradient-Flow Analysis

A new theoretical paper analyzes the rank-accuracy trade-off in Low-Rank Adaptation (LoRA) fine-tuning from a gradient-flow perspective. It rigorously derives the dynamical system equations for LoRA updates and establishes closed-form relationships between the update rank (r) and final model accuracy for specific loss functions. This work provides the mathematical foundation explaining why even rank-1 LoRA updates can achieve accuracy comparable to full-parameter fine-tuning, a phenomenon previously only observed empirically.