New audit reveals LALM judges cheat on speech evaluations
Six leading LALMs fail speech tasks when tested rigorously - here's why
A team from the University of Tokyo led by Joonyong Park has exposed critical reliability issues in large audio-language model (LALM) judges used for speech evaluation. Their paper, submitted to arXiv on July 15, 2026, audits six leading models across three common evaluation protocols: feature-blueprint judging, reference-conditioned judging, and pairwise A/B comparisons.
The research reveals that several models, including Qwen3-Omni-Thinking, exhibit what the authors call 'protocol-level shortcuts'—relying on structured text descriptions or reference data rather than analyzing the actual audio. In feature-blueprint judging scenarios, incorrect specialist labels caused five of the six models to drop emotion accuracy scores to 0.10 or below. In A/B comparison tests, Qwen3-Omni-Thinking often selected the same option regardless of order swaps, suggesting it wasn't actually evaluating the content. The findings indicate that high aggregate agreement scores with human ratings may mask fundamental flaws in how these models process speech data when the evaluation protocol itself provides shortcuts.
- Six LALMs (including Qwen3-Omni-Thinking) rely on protocol-level shortcuts instead of analyzing audio directly
- Emotion accuracy dropped to 0.10 in five models when given incorrect specialist labels in feature-blueprint judging
- Qwen3-Omni-Thinking selected the same slot regardless of order in A/B comparisons, suggesting flawed evaluation
Why It Matters
This audit reveals LALM judges may produce misleadingly high scores, threatening reliability of automated speech assessment in critical applications