AI Writes Your Code Fast, But Its Safety Checks Barely Catch Bugs
Most AI mistakes sail right past the tests meant to catch them.
Almost every big company now lets AI tools like ChatGPT or Copilot draft chunks of their software. The safety net is supposed to be automated tests — small programs that run the code and confirm it gives the right answer. This study asked a simple but uncomfortable question: when AI writes both the code and the tests, do the classic checking methods still work? The researchers tried three popular ones: coverage tests (did we run every line?), branch tests (did we try every path?), and mutation testing (deliberately break the code and see if the tests notice). They ran five AI models across four test suites and collected over 6,000 buggy programs.
The results were grim in a specific way. Most mistakes AI made were easy to catch — typos, missing steps, obvious slips. But the dangerous bugs, the ones that only appear on rare inputs, were hard even to trigger. And here's the real shocker: even when a bug did run, the detection rate stayed near zero. The reason is the "oracle" — the bit of the test that judges pass or fail. AI-generated oracles kept saying "looks fine" to broken code.
The practical stakes are high. AI-written code is already inside banking apps, HR systems, hospital scheduling tools, and the websites you use daily. If the automatic check wrongly says "all good," that bug ships to real customers — wrong balances, missed appointments, leaked data. The study also found that mutation testing, the fanciest and most expensive method, only barely beat the cheap coverage methods. That's like paying for a luxury car to get a slightly smoother ride to the same place.
The takeaway isn't "stop using AI to code." It's that someone still has to think about what the right answer actually is. The researchers found that tests written with extra hints about the goal caught more bugs, but even then the improvement was modest. Human judgment, for now, is the part you can't automate away.
- Researchers tested 5 AI models and 6,000+ buggy programs: standard software tests often miss AI-made mistakes entirely.
- Detection rates were near zero because the pass/fail check itself was written by AI and kept approving broken code.
- The fanciest test method, mutation testing, only slightly outperformed cheaper ones — so its high cost may not be worth it.
Why It Matters
Companies shipping AI-written code still need humans to double-check what the right answer looks like.