Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites

A research team has introduced STING, a framework that challenges the reliability of popular benchmarks like SWE-bench used to evaluate AI coding agents. The core problem is that benchmarks rely on regression test suites to validate AI-generated code patches. If these test suites are weak, they can accept "plausible yet semantically incorrect" patches, artificially inflating the reported success rates of models from companies like OpenAI, Anthropic, and others. STING uses a technique called mutation analysis, creating semantically altered variants of the correct patch. If a variant passes the existing tests, it reveals a gap in test coverage. These gaps then guide the automated generation of new, focused tests to strengthen the suite.

When applied to SWE-bench Verified, the results were striking. STING found that 77% of benchmark instances contained at least one surviving variant that passed the original tests but was incorrect. The framework generated 1,014 validated new tests across 211 instances, boosting line and branch coverage by 10.8% and 9.5%, respectively. The most significant impact came from re-evaluating leading AI repair agents with these strengthened tests. Their success rates dropped by 4.2 to 9.0 percentage points, revealing that a substantial portion of their previously celebrated "solutions" were exploiting test weaknesses rather than correctly solving the problem. This exposes a critical flaw in how AI coding prowess is currently measured and reported.