When AI Helpers All Agree, They Can Still Be Wrong — New Fix

Many of the AI tools you use now work like a panel rather than a single expert. Instead of one chatbot answering, several AI models (helpers that generate text) each write an answer, and the system keeps whatever most of them agree on. It sounds sensible: if three AIs say the same thing, it's probably right. This new paper shows that assumption is dangerous. Because these models were trained on much of the same material, they tend to make the same mistakes. A whole panel can confidently repeat one false claim — or all miss the one correct specialist fact.

To fix that, the team built a filter called C-MoA. For every individual claim in a long answer, it measures how strongly the other AIs actually support it, then sets a cutoff line and throws out anything below it. On long-form writing, the share of kept statements that were genuinely accurate jumped from roughly 41% to 75%. It also held up on a set of medical answers checked by human doctors, and it kept working on new topics without needing to be retrained.

The catch: on short answers — quick one-line responses — the filter was close to useless, basically guessing. Agreement is cheap when there's almost nothing to say. So the team added a second layer, CONTRA-MoA, which runs a kind of blinded near-miss contest: it asks whether a slightly different version of the claim would have sounded just as convincing, and it removes one AI from the panel at a time to see if the claim survives. That cut the number of false medical claims in half while staying 94% accurate — but only when the checking AI had real domain knowledge. With a checker relying on memory alone, performance collapsed to a coin flip.

The practical message: agreement between AIs is not proof, and better fact-checking requires a genuinely knowledgeable verifier, not just more AI voices.