SynIB boosts multimodal AI by targeting cross-modal synergy directly

A central challenge in multimodal learning is capturing synergy: task-relevant information that only emerges from combining multiple modalities, not from any single one. Most existing approaches tackle this by designing larger or more complex fusion architectures. In a new paper, researchers from KU Leuven and MIT propose a complementary axis: shaping the training objective itself. They introduce the Synergistic Information Bottleneck (SynIB), a scalable objective grounded in information theory that directly targets synergy. During training, SynIB runs forward passes with one modality masked at a time and penalizes the model for remaining confident—since high confidence with missing information indicates reliance on unimodal cues rather than true cross-modal interactions. This forces the model to learn representations that genuinely require joint reasoning.

SynIB was validated on synthetic XOR tasks where ground-truth synergy is known by construction—standard training failed to recover the synergy, while SynIB succeeded. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D, SynIB improved accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%. These results demonstrate that optimizing the training objective can achieve gains comparable to architectural changes, without increasing model size or complexity.