New framework reveals which modality drives MLLM decisions

A team of researchers led by Vahidin Hasić, Chao Wang, and colleagues has developed **Counterfactual Modality Attribution (CMA)**, a novel framework designed to address a critical gap in the explainability of multimodal large language models (MLLMs). These models, which combine text and image inputs for high-stakes decisions, often mask unsafe reasoning or shortcut learning by producing correct outputs without relying on the intended evidence source. CMA introduces a modality attribution score derived from cooperative game theory (Shapley values) and generated counterfactuals—image-only, text-only, and joint inputs—to quantify the contribution of each modality to a prediction.

The framework was tested on controlled synthetic benchmarks with known ground truths, where it correctly identified the decision-driving modality in **98% of cases**, significantly outperforming existing baselines. It was also evaluated on a real-world multimodal clinical dataset, demonstrating its practical utility in high-stakes environments. By establishing modality attribution as a complementary dimension of explainability, CMA provides a principled tool for auditing MLLMs, uncovering failures in cross-modal reasoning that predictive accuracy alone cannot detect. This work is detailed in their arXiv paper (arXiv:2608.00076) and represents a step toward safer, more transparent AI systems.