All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation

A team from National Taiwan University, led by Leonardo Haw-Yang Foo, has published a critical diagnostic study of Large Audio-Language Models (LALMs), revealing that many current benchmarks are fundamentally flawed as measures of true auditory perception. The researchers introduced a framework with two key axes: 'text prior,' which measures how answerable a test item is from text and general knowledge alone, and 'audio reliance,' which assesses actual dependency on the acoustic signal. By evaluating eight different LALMs across three standard benchmarks, they found that models retain 60-72% of their full audio scores even when no audio input is provided. Furthermore, among the items that did require audio, only 3.0-4.2% needed the complete audio clip; the vast majority could be answered correctly using just localized audio fragments.

These findings challenge the assumption that high benchmark scores equate to robust audio understanding. The paper, submitted to arXiv on April 27, 2026, concludes that many current evaluations inadvertently measure a model's ability to leverage text priors and general knowledge rather than its capacity to process acoustic signals. The researchers provide practical guidelines for improving evaluation reliability and benchmark design, urging the community to develop tests that truly require auditory comprehension. This work has immediate implications for developers and researchers working on audio AI, as it suggests that reported performance gains may be inflated and that more rigorous evaluation methods are needed to ensure models are genuinely listening, not just guessing from context.