RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing

A team of researchers, including Xuefeng Li, Nir Ben-Israel, and others from academic and industry backgrounds, has published a new paper introducing RepoMod-Bench. This benchmark is designed to rigorously evaluate AI coding agents on the complex task of modernizing entire code repositories, moving beyond simple code completion. The core innovation is an implementation-agnostic testing paradigm that treats the original source repository as the ground truth. By focusing on projects with standardized interfaces, the benchmark can verify functional equivalence between source and AI-generated target code using a black-box test suite, which is hidden from the agents to prevent test-driven overfitting—a major flaw in previous evaluation methods.

RepoMod-Bench is substantial, comprising 21 real-world repositories across 8 programming languages, totaling 1.6 million lines of code and 11,616 tests, with project sizes ranging from 14 to 211,000 lines. When the researchers evaluated four leading AI agent configurations, they uncovered a critical scaling problem: average pass rates plummeted from 91.3% on projects under 10K lines of code to just 15.3% on projects exceeding 50K lines. This dramatic collapse demonstrates that while current AI agents can handle small-scale refactoring, they struggle significantly with the complexity and interdependencies of large, real-world codebases. The findings establish a clear, measurable frontier for AI-assisted software engineering, indicating that achieving reliable, autonomous repository-level modernization is still a significant open challenge for the field.