New Study: Teams of AI Chatbots Aren't Smarter Than One Good Idea
You may be paying 10x the AI cost for the exact same answer.
Many companies now run "multi-agent systems" — several AI assistants (agents are AI that can take actions, not just chat) working on one problem together, like a project team. The pitch is that more AI minds produce better answers. A new research paper tested that claim directly, and the result is a warning for anyone paying for it.
The team built MASTraceBench, a testing tool that follows the whole conversation rather than only grading the final answer. It records every proposal each AI makes, how those proposals change as agents talk, and how much computing power — measured in tokens, the units AI companies bill by — the whole debate costs. They ran it across six tasks, some where the AIs cooperate and some where they compete.
The pattern was consistent: the group's final answer rarely beat the strongest opening idea from any single AI. What usually happened was that weaker suggestions got pulled up toward the best one. The best idea itself was almost never made better, and sometimes it got watered down by compromise. Think of a meeting where the smartest person speaks first, everyone else agrees, and you all leave with the same plan — after two hours of talking.
To fix this, the authors propose CLEARS. Instead of trading whole proposals back and forth, it breaks each proposal into individual claims and has the other AIs grade and merge those small pieces. That way good details survive instead of being averaged away. CLEARS preserved or improved the strongest opening idea more often, and scored the highest "collaboration gain" on five of the six tasks.
The takeaway for buyers of AI tools: a swarm of agents isn't automatically better than one strong model plus good review. If your AI bill is climbing because you stacked agents together, it may be worth asking whether the answers are actually improving.
- A new benchmark, MASTraceBench, grades not just AI teams' final answers but how they got there — and what it cost in tokens.
- Across six tasks, the group's answer almost never beat the single best first proposal; strong ideas were rarely improved and sometimes got worse.
- The researchers' alternative method, CLEARS, evaluates ideas claim by claim and delivered the biggest collaboration gain on five of six tasks.
Why It Matters
Stacking multiple AI agents may add cost without adding accuracy — check before you pay more.