Beyond Localization: Recoverable Headroom and Residual Frontier in Repository-Level RAG-APR

A new research paper titled "Beyond Localization: Recoverable Headroom and Residual Frontier in Repository-Level RAG-APR" examines the fundamental limitations of AI-powered code repair systems. Researchers from multiple institutions tested three leading repository-level RAG-APR (retrieval-augmented generation for automated program repair) paradigms—Agentless, KGCompass, and ExpeRepair—on the SWE-bench Lite benchmark. Using a sophisticated protocol with Oracle Localization (perfect bug identification), within-pool Best-of-K sampling, and controlled context probes, they discovered that even with ideal conditions, these systems couldn't surpass a 50% success rate.

The study reveals that while better bug localization improves all three systems, the gains quickly saturate. Extra candidate diversity helps within sampled 10-patch pools, but that headroom is limited. Under fixed interfaces, most informative added context conditions still outperformed their matched controls, suggesting evidence quality matters. However, the common-wrapper check showed different system responses: KGCompass and ExpeRepair maintained gains under a common wrapper, while Agentless was more sensitive to builder choice. Most strikingly, prompt-level fusion left a large residual frontier—the best fixed probe added only 6 solved instances beyond the native three-system union.

Overall, the research demonstrates that stronger localization, bounded search, evidence quality, and interface design all shape repository-level repair outcomes, but current approaches face inherent limitations. The findings suggest that simply improving existing techniques won't be enough to achieve high success rates in automated code repair, pointing to the need for more fundamental architectural innovations in AI programming assistants.