Study: Smaller AI Can Catch Bigger AI's Coding Mistakes

AI coding tools can now write real software for companies. The problem: they sometimes hand back code that looks finished and reads confidently, but quietly skips something important. Because these AI agents work in long, messy steps, a human reviewer can't easily spot what got left out. So the researchers asked a practical question: can a smaller, cheaper AI reliably act as the reviewer for a bigger, smarter one?

They analyzed 411 real coding sessions and 101 controlled test cases. At first, just showing the reviewer structured notes about what the code did made it both catch more genuine defects and reject more working code — a trade-off. But when they handed the reviewer official execution evidence (basically, proof the code actually ran and passed its tests), things improved sharply. Five of six reviewers caught more real problems while wrongly rejecting less. Two of them judged every single case correctly.

Surprisingly, the reviewer's size didn't reliably predict how well it did. Bigger was not consistently better. The deciding factor was 'groundability' — whether the reviewer had hard facts to stand on, not opinions or summaries.

In the real world, official test results usually aren't available. So the team tried a fallback: automatic error checks plus freshly generated tests that first fail on the broken version of the code. On held-out cases, this approach caught roughly 76-80% of genuine defects. But it wrongly rejected good code about two-thirds of the time — mostly when messy, unresolved cases landed in the reviewer's lap. Their conclusion: weak AI reviewers can work, but producing trustworthy checks without official tests is still the hard part.