New study reveals how multimodal LLMs route audio and visual information

A new paper titled 'From Senses to Decisions' investigates the internal information flow of auditory and visual tokens within Audio-Visual Large Language Models (AVLLMs). By tracing how these signals route through networks like Qwen2.5-Omni and Video-SALMONN2 Plus (at 3B and 7B scales), the authors discovered two distinct processing patterns: sequential pathways for single audio-visual videos (similar to VLMs), and parallel streams for multiple interleaved audio-visual items. The study also found that the contribution of each modality scales proportionally to the task's reliance on that input.

Perhaps the most striking finding is that audio-visual tokens can be discarded once their information has been transferred to the LLM's internal representations—without harming prediction accuracy, and in some cases even slightly improving it. This ‘pruning’ effect generalizes across multiple tasks and datasets, opening the door to more efficient inference in multimodal systems. The researchers hypothesize these flow structures emerge from architectural constraints and training dynamics. The work delivers the first coherent picture of how AVLLMs orchestrate sound and sight internally, laying groundwork for improved interpretability, design, and efficiency in next-generation multimodal LLMs.