AI Coding Tools' Bug-Fix Scores Are Misleading, Study Finds

A team of software-engineering researchers looked at how AI systems that fix broken code — known as automated program repair, or APR (software that finds and patches bugs on its own) — are graded. Almost every paper tests its system on the same two well-known test sets, Defects4J and SWE-bench, and then reports something like "we fixed 30% of the bugs." The problem: those two test sets don't describe what the AI was actually asked to do.

In one setup, the AI is handed the exact file and even the exact line where the bug lives. In another, it must search an entire codebase, run tests, read error messages, and try again after failing. Those are wildly different jobs, yet both get reported as a single percentage. The researchers documented how much the details vary across systems: what information was given before the AI started, which tools it could use mid-task, what feedback it received after a failed attempt, how the final patch was checked, and how much computing power it was allowed to burn.

The practical consequence is that a higher score doesn't necessarily mean a better tool. A system that scores 40% on the easy version of a test may be far weaker than one scoring 25% on the hard version. Companies picking AI coding assistants, managers tracking productivity gains, and researchers claiming progress can all be misled by numbers that look comparable but aren't.

The team's proposed fix is a machine-readable checklist, essentially a standard label that every result would carry, spelling out all of those conditions. That way, anyone can see whether two systems were genuinely playing the same game. The catch: it's a voluntary proposal, not a rule. Nobody is forced to publish the label, and vendors have an incentive to keep their best-looking numbers. So the advice stands for now — when you see an AI coding claim, ask what help it got.