RoCo-ACE: New method injects knowledge into MLLMs without catastrophic forgetting
Keeps retention near base model while achieving best injected-knowledge accuracy across six benchmarks
Updating pretrained multimodal large language models (MLLMs) with new factual or domain-specific knowledge—known as knowledge injection—often causes drift: the model loses performance on non-updated behaviors because it tries to fit full authoritative answers. Existing online distillation mitigates drift by training on model-generated rollouts, but its uniform reference-conditioned supervision is coarse—it under-emphasizes reference-supported rollout tokens and only indirectly supervises omitted facts.
RoCo-ACE tackles this with a two‑pronged approach: (1) Rollout-Conditioned (RoCo) weighting uses the contrast between reference-free and reference-conditioned likelihoods on the same rollout to redistribute distillation weight toward reference-supported tokens. (2) ACE adds sparse reference-side anchored correction for authoritative anchors that the rollout omitted, without requiring full-answer imitation. Tested across three knowledge-injection settings, six retention benchmarks, and multiple base models, RoCo-ACE consistently beats all compared methods on injected-knowledge accuracy while keeping retention performance nearly identical to the unmodified base model. This makes it a practical drop‑in technique for safely evolving deployed MLLMs with new facts.
- RoCo-ACE uses same-rollout likelihood contrast to reweight distillation toward reference-supported tokens
- ACE adds sparse correction for facts omitted from rollouts, avoiding full-answer imitation drift
- Achieves best injected-knowledge accuracy across 3 settings and 6 retention benchmarks, with retention near base model
Why It Matters
Enables safe, precise updates of large multimodal models with new domain knowledge—critical for enterprise AI that must stay current without regressing.