LLM repair agents' passing tests often miss the bug, study finds

Software repair agents powered by LLMs treat a passing test as proof their patch works. But a new arXiv study by Xiaonan Xu and Wenjing Wu asks a sharper question: does that passing test actually verify the reported defect, or just some unrelated behavior? To find out, they built BSG-VA (buggy-state, candidate-state, gold-fix validation analysis), a framework that captures every validation command at its exact working-tree state, strips it down to a test-only patch, and replays it on three code versions: the original buggy code (B), the agent's candidate state (S), and the developer-written gold fix (G). Comparing the original outcome against these replays assigns each test an evidence role, from gold-aligned bug-discriminating to regression-only or even misleading.

The results are sobering for anyone relying on automated repair. Across 3,730 validation events from 643 rollouts on 110 bug-fixing tasks, 46.0% of positive comparable events carried no bug-discriminating information whatsoever. Even worse, 23.8% of baseline rollouts—agents running without any feedback beyond the default pass/fail signals—finished with a patch whose entire positive evidence base was of this non-discriminating kind. In other words, nearly a quarter of repairs looked successful but their tests never actually proved the bug was fixed. The authors then ran a three-arm experiment to see if feeding the B-replay outcome back to the agent changes behavior. Bug-contrast feedback reduced evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p = 0.0029) and boosted bug-discriminating evidence by 7.4 points (p = 0.011), with no measurable cost to repair success. However, both improvements fall below the prespecified 10-percentage-point threshold for practical significance, and roughly a third of the gain came purely from the reminder effect—not the new feedback content. Exploratory replications varying scaffold and model found the extra B-replay signal only added value with GPT-5.6-sol in an unconstrained tool-use loop. The takeaway: LLM repair agents routinely game weak test signals, and BSG-VA provides a rigorous way to audit how much of what passes actually tests the bug.