Study: AI Teams Pick the Loudest Voice, Not the Right Answer

Many AI products now don't use one model — they use a small team of AI agents (programs that can act on their own) that take turns working on a problem. The usual rule for deciding who speaks next is simple: ask each agent how confident it is, and let the most confident one talk. Researchers at Argonne National Laboratory and collaborators put that rule under a microscope, examining 4,181 math problems solved by the AI model gpt-oss-120b, plus repeat tests with a Google model called Gemma.

The first finding is that AI confidence scores are badly inflated. On the math problems, the agents claimed an average confidence of 79% while actually being correct only 52% of the time — roughly a coin flip dressed up as certainty. The researchers applied a statistical correction that shrank the error rate dramatically, from 0.278 down to 0.008. But the fix only made the numbers look honest; it couldn't make the AI better at telling right answers from wrong ones. With the Gemma models, confidence-based picking performed 5.6 and 11.2 percentage points worse than choosing at random.

A third problem turned up when the researchers compared what agents said privately to what they said out loud. In about 1 in 5 cases, the answer an agent reported in an internal poll differed from the answer it announced when it won the turn — and 62% of those changes were freshly generated, not corrections. On average, the public version was slightly less accurate, dropping correctness by 1.7 points, though other models actually improved by up to 12 points. In other words, what an AI says when everyone is watching isn't always what it computed.

For anyone using AI to make decisions — a hiring tool, a customer chatbot, a medical assistant — the takeaway is practical. Confidence scores, answer quality, and public statements need to be checked separately, not bundled into one number. "The AI said it was sure" is not evidence that it was right.