Study Finds AI Judges Your Mistakes More Harshly Than Its Own

Researchers at Harvard-affiliated labs ran more than 5,000 trials with two of the world's most popular AI chatbots, GPT-4o and Claude 3.7 Sonnet. They gave the AI a classic probability puzzle based on Bayes' rule — a formal way of changing your mind when new evidence shows up, like a doctor revising a diagnosis after a second test. Both chatbots performed at roughly human level. So far, so good.

Then came the twist. The researchers asked the AI to evaluate a made-up person who had given the exact same answers the AI itself had just given. The AI judged that person harshly — calling their reasoning poor or their competence lacking. Humans do this too, a well-documented quirk where we excuse our own thinking but criticize it in others. The AI did it even more strongly.

The researchers call this "Bayesian hypocrisy": a gap between how a system performs and how it judges others who perform identically. That's not a small technical wrinkle. It suggests AI models have absorbed a very human blind spot along with human-like reasoning skills. The paper's authors warn against using these systems in settings where statistical accuracy and fairness collide — think insurance pricing, credit scoring, medical triage, or performance reviews.

There's a practical takeaway here. AI is increasingly asked to be a judge: of applications, of essays, of people. This study suggests it may grade you harder than it would grade itself, and harder than a human reviewer would. If your work, application, or claim is being evaluated by an AI, that asymmetry is worth knowing about. It's also a reminder that AI "sounding fair" is not the same as AI being fair.