AI Teams Can Give Right Answers for the Wrong Reasons
A correct answer doesn't prove the AI teamwork actually worked.
More and more AI products now run as teams: several AI programs pass tasks to each other, look things up, double-check each other's work, and hand you one final answer. The idea is that teamwork makes AI more reliable. A new paper by researcher Zhengye Han asks a blunt question — if the team gives you the right answer, can you safely assume the teamwork actually happened the way it was supposed to?
The answer is no. Han studied what he calls collective mechanisms: the rules for who gets told what, whose information is trusted, and who is allowed to act. He replayed real AI runs three ways — unchanged, with one rule deliberately broken, and with it repaired. Breaking a rule often left the final answer completely correct. So a good result tells you almost nothing about whether the process was sound. When an AI was used to inspect the internal logs, it caught far more broken rules than when it only saw the public output.
The catch: AI inspectors are overconfident. Looking at the same records, a generic prompt often claimed certainty the evidence did not support — a problem the stricter contract-style prompts mostly avoided. Worse, a checker that performed nearly perfectly on the study's own tests got noticeably worse on a differently built system. You cannot assume a tool that works on one AI setup will work on yours.
Why this matters in practice: AI agents are starting to book appointments, move money, fill out forms, and summarize sensitive documents. If they get those tasks right by luck — or by skipping a safety check, a source, or a teammate's warning — nobody notices until something goes wrong. Ask for logs you can inspect, and treat 'it worked' as different from 'it worked correctly.'
- Researchers deliberately broke the teamwork rules inside AI teams, and the final answer was still often correct — a good result can hide a broken process.
- Inspecting the AI's internal records caught far more problems than looking at the final answer; generic prompts claimed certainty the evidence didn't support.
- A checker that scored near-perfect on one AI system performed worse on a differently built one, so don't assume these tools transfer.
Why It Matters
If your AI helpers get things right by accident, you won't know — so demand records, not just results.