Curvature-Guided Mixing prevents catastrophic forgetting in MLLMs

Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting, where the model loses its general capabilities. Existing model merging methods rely on heuristics or sub-optimal objectives, leaving room for improvement. Researchers introduce Curvature-Guided Mixing (CGM), a theoretically grounded framework that formulates a joint optimization objective. By leveraging a second-order (Hessian) approximation of the loss landscapes, CGM analytically derives an optimal, closed-form soft mixing ratio that intelligently blends parameters based on their relative task-specific curvatures. This ratio determines how much of the pre-trained versus fine-tuned model to retain for each parameter, effectively preserving general knowledge while adapting to new tasks.

To further enhance efficiency, the paper also introduces CGM†, a robust hard mixing variant that performs sparse parameter selection guided by a novel, curvature-aware score. This variant selects only a subset of parameters to update, avoiding unnecessary modifications. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks demonstrate that both CGM and CGM† consistently improve the trade-off between task specialization and general knowledge retention, outperforming existing model merging methods. The code is publicly available, and the work has been accepted to ECCV 2026.