AI Code Fixers Pass Every Test — Then Leave Security Holes

AI coding agents are quickly becoming a normal part of how software gets built. You describe a bug, the AI writes a fix, the team's automated tests run green, and everyone moves on. A new study from researchers in Finland and Pakistan suggests that green light may be misleading. They looked at 1,030 repair attempts made by seven different AI agent systems using GPT-4o-mini, an affordable OpenAI model, across two collections of real security bugs.

Out of those attempts, the researchers manually confirmed 170 cases where the AI's fix passed every check but still left the code vulnerable to hackers. They call these "silent failures" — nothing crashes, no alarm goes off, but the door is unlocked. The most common problem, 48.2% of cases, was omission: the AI fixed the visible bug but never added the security protection that was needed. Another 30.6% actually introduced a brand-new weakness that wasn't there before. The remaining 21.2% added a defense that was too weak to matter.

Perhaps the most uncomfortable finding is that the normal safety nets didn't work. Passing test suites didn't catch these issues, and using a second AI as a reviewer didn't catch them either. Different agent frameworks also produced surprisingly similar insecure fixes, hinting that the underlying AI model, the instructions it was given, or how the task was framed may push it toward the same blind spots.

For anyone whose company is rolling out AI coding tools, this is a reminder that "the tests passed" is not the same as "this is safe." The researchers argue teams need checking methods that look beyond whether the code simply works, and inspect everything the AI produces. The takeaway isn't to stop using these tools — it's to keep a human with security know-how in the loop before anything ships.