ReCoLoRA lets LLMs learn task sequences without forgetting previous ones
A new recursive consolidation method beats LoRA on continual learning benchmarks
Parameter-efficient fine-tuning methods like LoRA struggle with continual learning: stacking low-rank updates on frozen weights causes new tasks to overwrite previous ones. ReCoLoRA (Recursive Consolidation of Low-Rank Adapters) tackles this head-on. The method initializes adapters from a randomized SVD of the pretrained weight, selects per-layer effective ranks using an elbow criterion, and adapts the principal subspace before opening residual capacity. Crucially, before each new task, ReCoLoRA re-decomposes the current effective weight into three components: a frozen residual, a slowly updated principal component, and a fresh adapter. This recursive consolidation ensures every new task starts from a model that has already absorbed its predecessors.
Tested on a six-task continual GLUE sequence over four 7-8 billion parameter backbones, ReCoLoRA outperformed rank-swept LoRA, PiSSA, AdaLoRA, and DoRA on three out of four models—all while training fewer parameters. An oracle-routed task-bank variant set an upper bound under full task isolation. The results demonstrate that spectrum-aware recursive consolidation provides a practical, parameter-efficient path to continual learning for large language models, potentially enabling applications like lifelong personalization, continual instruction tuning, and domain adaptation without catastrophic forgetting.
- Uses randomized SVD of pretrained weights and an elbow criterion to determine per-layer effective ranks
- Recursively decomposes the effective weight before each new task to preserve prior knowledge
- Achieves best average score on 3 of 4 7-8B backbones against LoRA, PiSSA, AdaLoRA, and DoRA with fewer parameters
Why It Matters
Enables efficient continual learning for LLMs, reducing catastrophic forgetting without full retraining or storing all past data.