ECA: Efficient Continual Alignment prevents catastrophic forgetting in image-to-text generation

A team of researchers has proposed ECA (Efficient Continual Alignment), a new exemplar-free incremental learning (IL) approach for open-ended image-to-text generation (OpenITG). The method addresses a practical scenario where visual data categories shift over time—for example, a model trained on animals later seeing vehicles. Unlike prior continual learning work, ECA focuses on continually adapting the alignment module within pre-trained vision-language models (VLMs) without accessing raw data from previous tasks. The key challenge is acquiring new task-specific features while minimizing interference with established cross-modal representations.

ECA employs three core mechanisms to achieve this. First, a Mixture of Query (MoQ) module adapts task-specific query tokens to handle new visual concepts. Second, Fisher Dynamic Expansion (FeDEx) dynamically grows model capacity based on a Fisher Information Matrix (FIM) metric, deciding when to add new parameters. Third, an embedding dictionary with Dictionary Replay (DR) retains distilled knowledge from past tasks, preventing catastrophic forgetting. The authors constructed four new IL benchmarks reflecting real-world data shifts. Experimental results showed ECA significantly outperforms baseline methods in both forgetting reduction and generation quality. The code and benchmarks are open-sourced.