CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows

Researchers from Concordia University have released CI-Repair-Bench, a repository-aware benchmark for automated patch validation that uses real Continuous Integration (CI) workflows from GitHub Actions. The benchmark addresses a critical gap in program repair research: existing benchmarks focus on source-code-level test failures and ignore the complex, multi-stage CI pipeline. CI-Repair-Bench contains 567 CI failure instances drawn from 103 open-source repositories, each failure categorized into one of 12 distinct error types such as formatting violations, dependency conflicts, environment misconfigurations, and test flakiness. Repairs are validated by re-running the original CI workflow, ensuring correctness is measured in a realistic, production-like setting.

The benchmark's evaluation of state-of-the-art LLM-based repair systems reveals stark limitations. While automated repair excels at simple, tool-enforced issues like linting and formatting (often hitting >90% success), it struggles with environment-specific failures such as missing runtime dependencies or configuration mismatches. The best-performing LLM (likely GPT-4) achieved only an 18.9% overall repair success rate. These results underscore how far autonomous CI repair has to go, and provide a rigorous foundation for future research. The paper also includes a reference CI repair workflow that parses logs, localizes faults, and generates candidate patches – giving practitioners a starting point for building real-world CI debugging tools.