DCASE 2026 ADQA benchmark reveals audio-language model limits at 58% accuracy

DCASE 2026 Task 5 tackles a critical flaw in audio-language model evaluation: many existing benchmarks can be solved using textual priors alone, without actually understanding audio content. To fix this, the authors built an Audio-Dependency Filtering (ADF) pipeline that combines silent-audio probing, per-option perplexity scoring, an LLM commonsense check, and human review. From an initial pool, they selected 3,000 items that truly require audio input—spanning music, speech, and environmental sounds—forming the ADQA-Bench evaluation set. This design ensures that models must listen to the audio to answer correctly.

The inaugural competition attracted 14 teams and 36 submissions across two tracks (up to 100B parameters and under 10B). The top score overall was 58.33% by a Chung-Ang University ensemble of MOSS-Audio-8B-Thinking and Qwen3-Omni-30B. In the sub-10B track, a MOSS-only configuration from the same team led at 57.30%. Notably, evaluation accuracy on the hidden test set—designed to be harder than the development split—dropped by an average of 11.91 percentage points. The most common technical building blocks included MOSS-Audio-8B-Thinking as backbone (13 of 36 submissions), LoRA fine-tuning on AudioMCQ-StrongAC, and reinforcement learning objectives such as GRPO and GDPO. All systems failed on the same 233 items, indicating a hard core of audio-dependent questions that current architectures cannot solve.