Researchers propose CMCDR to fix AI's modality drift problem
New method CMCDR from Zhen Zhang et al. stops AI from forgetting how to use data modalities
A team of researchers from the University of Electronic Science and Technology of China and other institutions has developed Continual Modality Contribution Drift Regularization (CMCDR), a novel approach to address a critical but overlooked challenge in multimodal continual learning (MMCL): Modality Contribution Drift (MCD).
MCD occurs when AI models, while incrementally learning new tasks, unknowingly shift how they rely on different data modalities (e.g., text vs. images), leading to degraded performance on previously learned tasks. Current MMCL methods focus on aligning representations or preserving semantic similarity but fail to stabilize modality-specific contributions. The proposed CMCDR introduces a diagnostic framework using modality-subset interventions to quantify and regularize these contributions. It comes in two versions: one that uses stored old samples (replay-based) and another that relies solely on current-task data (replay-free). Both methods empirically outperform existing approaches in multimodal class-incremental learning and continual visual question answering.
- Introduces 'Modality Contribution Drift' (MCD), a new metric to quantify how AI models change their reliance on different data modalities over time.
- Proposes CMCDR with replay-based and replay-free variants to stabilize modality contributions during incremental learning.
- Validated on multimodal benchmarks, achieving state-of-the-art performance in class-incremental learning and continual VQA.
Why It Matters
Prevents AI systems from degrading over time as they learn new tasks, ensuring consistent performance across modalities.