New IndicContextEval benchmark tests if AudioLLMs actually use context, not just memory

Audio large language models (AudioLLMs) can transcribe speech conditioned on text prompts like domain descriptions or entity lists, but it's unclear if they truly leverage that context or just rely on pre-trained knowledge. Existing benchmarks don't test this because they use fixed prompts and rarely include explicit contextual inputs. To fill this gap, the team behind IndicContextEval built a 56-hour multilingual benchmark of natural speech from 555 speakers across 8 Indian languages (Hindi, Tamil, Telugu, etc.) and 23 professional domains. The dataset includes metadata, natural-language descriptions, entity lists in English and native script, and adversarial prompts with incorrect entities to stress-test context utilisation.

Using a 7-level prompting framework that progressively adds contextual signals, the researchers evaluated five state-of-the-art AudioLLMs. Results showed substantial differences in how models use context: some benefited significantly from entity lists, while others barely improved or even degraded with adversarial prompts. This reveals that current AudioLLMs have inconsistent contextual grounding, and simple transcription accuracy metrics hide these deficiencies. The work, accepted at Interspeech 2026, provides a new evaluation standard for building more reliable speech AI systems, especially important for low-resource Indic languages where accurate transcription with domain context is critical.