Researchers unveil PAIR-Bench to evaluate AI code repair

Researchers from the University of Texas at Dallas and the University of California, Irvine have developed PAIR-Bench, a next-generation benchmark designed to evaluate how large language models (LLMs) improve buggy or incomplete code through *progressive feedback*. Unlike traditional benchmarks that only measure binary functional correctness (pass/fail), PAIR-Bench introduces a structured feedback protocol called *progressive hinting*, which controls both the *failure-region* (what the feedback targets) and *hint-depth* (how much repair-relevant information is revealed). This allows the benchmark to assess not just whether a model eventually fixes the code, but *how* it gets there—tracking refinement trajectories, generalization beyond hints, and preservation of already-correct behavior.

The team argues that current benchmarks fail to capture the nuanced capabilities of modern LLMs in iterative development workflows. PAIR-Bench addresses this by introducing *progressive metrics* that evaluate repair trajectories rather than just final outcomes. The benchmark groups failing tests into *failure scenarios* and gradually reveals guidance, from coarse symptoms to implementation-level fixes. This design enables a finer-grained assessment of an LLM’s code-improvement ability—measuring whether it repairs targeted failures, generalizes beyond hints, preserves correct behavior, and how much assistance it requires. The findings suggest that PAIR-Bench could become a new standard for evaluating AI-assisted software engineering tools.