TokenSwap reveals multimodal LLMs drop 19.6% on image-interleaved inputs

A new arXiv paper from researchers including Andong Hua, Igor Mordatch, Aleksandra Faust, and Yao Qin introduces TokenSwap, a technique that quantifies and reduces the modality gap in multimodal large language models (MLLMs). The modality gap is defined as the systematic performance difference when models are given semantically equivalent inputs in text-only versus image-interleaved formats. TokenSwap constructs such inputs by replacing textual concepts with semantically aligned images, creating sequences where visual tokens interleave with text tokens. Using this method, the team transformed text-based benchmarks like MMLU into TokenSwap-Bench for cross-modal evaluation.

Across 42 MLLMs, the results show a pervasive problem: moving from text-only to image-interleaved inputs reduces accuracy by an average of 19.6% (standard deviation 3.3%), with some models dropping as much as 47.4%. Notably, reasoning models are significantly more robust, showing only a 10.1% average gap compared to 25.5% for non-reasoning models. However, neither prompting strategies nor scaling training compute alone reliably closes the gap. The authors demonstrate that incorporating TokenSwap directly during training mitigates the issue while preserving strong performance on both text-only and vision-language tasks. This benchmark and mitigation approach provides a practical tool for evaluating and improving multimodal robustness.