Continual Learning Fails to Balance Stability in Medical VQA

Deploying medical visual question answering (MedVQA) systems in real-world clinical settings requires models that can adapt to new tasks without forgetting prior knowledge. Researchers led by Mai A. Shaaban from MBZUAI present a systematic evaluation of continual learning (CL) methods for heterogeneous MedVQA tasks, including classification, multi-label classification, detection, cell counting, and report generation. The study explores three key aspects: the ability of existing CL methods to mitigate catastrophic forgetting, their sensitivity to task ordering, and the evolution of low-rank adaptation (LoRA) parameters as new tasks are learned. The findings reveal that current CL methods struggle to maintain a stability-plasticity balance when tasks with fundamentally different objectives and supervision formats are interleaved.

The analysis shows that task ordering significantly impacts performance retention and forgetting rates, with some sequences causing complete loss of earlier task performance. Additionally, the study tracks LoRA weight drift, revealing distinct patterns under different CL approaches. The results highlight a critical limitation for clinical AI deployment, where models must continuously learn from new modalities and protocols without losing previously acquired diagnostic capabilities. The authors will release code and full experimental setups to facilitate further research. This work underscores the need for more robust continual learning strategies tailored to medical domains, where catastrophic forgetting could have serious patient safety implications.