MAP: Predicting attention to prune visual tokens speeds up MLLMs 3x

Multimodal large language models (MLLMs) like LLaVA-NeXT process thousands of visual tokens, making inference expensive. Visual token pruning reduces this cost, but it typically relies on attention maps from a fixed middle layer. Researchers Sun et al. found that the layer most responsive to a given question varies per sample, so a fixed layer is suboptimal—and computing attention from that layer already requires running several transformer layers, defeating much of the efficiency gain.

To fix this, the team proposes Middle-layer Attention Prediction (MAP). It uses Question Contrastive Teacher Selection to pick a sample-specific teacher layer by comparing attention under the original question versus a reference question, then distills that attention into a lightweight predictor that estimates token importance directly from multimodal input features. At inference, MAP combines predicted importance scores with a diversity criterion to prune visual tokens before the first language model layer, requiring no attention maps and staying compatible with other acceleration techniques. On ten benchmarks with LLaVA-NeXT-7B, MAP retains 97.5% of unpruned performance using only 5.56% of visual tokens, yielding a 3.09x end-to-end speedup.