AI Judges Rate Voices by Volume, Not Quality, Study Finds

AI is increasingly asked to be the judge. Instead of humans grading customer service calls, podcast auditions, language lessons or job interviews, companies hand the job to a speech-capable AI that listens and gives a score. This paper tested six of those AI judges to see whether they rate speech fairly — and found they don't.

The researchers changed only one thing at a time: how loud the recording was, how much content it contained, and how emotional the speaker sounded. The AI judges reliably gave higher marks to louder audio. They rewarded dense, content-rich speech more strongly than human listeners did. And they translated emotional delivery into an overall quality score, even when emotion had nothing to do with what was being judged. The researchers call this an "acoustic shortcut" — latching onto something that sounds impressive rather than something that actually matters.

The most troubling part is the explanation. Modern AI judges are supposed to tell you why they scored something a certain way. In this study, their written justifications almost never mentioned the loudness or tone that actually drove the decision. They leaned on the same vague praise phrases over and over. So a candidate could be marked down for a quiet microphone, and the feedback would blame something else entirely.

One important nuance: these biases showed up mainly when the AI compared two recordings side by side. When it scored each clip on its own, the problem was harder to spot — which means many real-world systems may be quietly affected. The team released their test recordings and tools so others can check their own AI judges. For anyone whose voice is being graded by software, the takeaway is simple: ask what the AI is actually measuring.