New Cheap LoRA cuts training time 10% and memory 15% with sparsity

Low-rank adaptation (LoRA) has become a go-to technique for fine-tuning large models without updating all parameters, but questions remain about whether its low-rank structure is optimal for generalization. A new paper by Cadenhead et al. challenges this by introducing sparsity-induced alternatives: Cheap LoRA (cLA) and its chained circulant variant c³LA. cLA trains only a single low-rank factor while keeping the other fixed—either deterministically or stochastically—effectively acting as a structured column-subspace restriction of full fine-tuning. The authors derive information-theoretic generalization error bounds, one of the first attempts to formally analyze such methods.

Empirically, the team evaluated 11 fine-tuning methods across 10 pre-trained models and 14 datasets, using loss landscapes and spectral analysis. Despite the sensitivity of fine-tuned models to choice of base model and data, cLA and c³LA remained competitive with parameter-matched baselines while cutting training time by up to 10% and peak GPU memory by up to 15%, even with a naïve sparse implementation. The theoretical and empirical generalization measures provide a more consistent, principled cost-effectiveness analysis than common heuristics. This work suggests that embracing sparsity—rather than adhering strictly to low-rank structure—can yield simpler, cheaper, and still performant fine-tuning.