AI Judges Are Inconsistent, New Study Finds

When AI does open-ended work, like writing a story or solving a complex problem, there's no single right answer. So, another AI often acts as a judge. But what if that judge is inconsistent? A new study tested AI judges on 378 actions from a game and found that when the same judge re-evaluated the same action, it changed its decision 3 to 16 percent of the time. That's like a teacher grading the same essay differently each time they read it.

The study borrowed methods from wargaming, where referees have been making judgment calls for centuries. They tested different ways of judging: from a fixed rulebook to a free AI judge to a panel of four AIs. The free AI judge was the most inconsistent, changing up to 16% of outcomes. Even swapping one AI model for another changed results by about 3 percentage points—a difference that's hard to confirm with just nine games.

Why does this matter? AI judges are increasingly used in real-world decisions: grading student essays, screening job applications, moderating online content. If the judge is inconsistent, people could get unfair outcomes. The study found that a panel of AIs changed 40 out of 378 outcomes, but the changes cancelled out, so the overall pass rate looked stable. That's dangerous because it hides the inconsistency.

The takeaway: before choosing which AI to use as a judge, decide on the judging procedure. And always report how often the judge disagrees with itself. This research is a warning that AI judging isn't as reliable as we might think—and we need better methods to ensure fairness.