FoLoRA fine-tunes AI without forgetting other skills

Fine-tuning a foundation model for a specific task (e.g., math) often degrades unrelated skills (e.g., code generation). Existing methods use static constraints or specialized initialization but fail to dynamically regulate the trade-off during training. A new paper from researchers at multiple institutions introduces Foundation Preserving LoRA (FoLoRA), an optimization framework that continuously scores each update direction by how much task utility it provides per unit of forgetting penalty. This is achieved through a first-order preservation condition and a generalized Rayleigh-quotient formulation that compares activations from the downstream task against pretraining-proxy calibrations.

FoLoRA constructs a spectral coordinate system from these scores and applies gated Adam updates: directions with low utility-to-penalty ratio are attenuated during training. To estimate the forgetting penalty without relying on a single proxy dataset, FoLoRA samples calibration data directly from the pretrained model, making it more robust. Experiments on math, code, and instruction-following tasks show FoLoRA achieves the strongest preservation-adaptation balance, improving target performance while retaining non-target capabilities better than baselines. The approach is compatible with LoRA-based fine-tuning, making it practical for large models.