SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark

A research team led by Boxi Yu has published a paper revealing significant inflation in AI coding benchmark performance, specifically targeting the popular SWE-Bench Verified leaderboard. Their new framework, SWE-ABS (Adversarial Benchmark Strengthening), demonstrates that approximately 20% of "solved" patches from top-30 AI coding agents are actually semantically incorrect but pass due to weak test suites that fail to expose their errors. This finding challenges the apparent saturation of the leaderboard, where the top system had achieved 78.80% success.

The SWE-ABS framework operates through a two-stage pipeline: first, coverage-driven augmentation uses program slicing to target untested code regions; second, mutation-driven adversarial testing synthesizes plausible but incorrect patches to expose semantic blind spots. On SWE-Bench Verified's 500 instances, SWE-ABS strengthened 50.2% of cases—a 25.1x improvement over prior methods—and rejected 19.71% of previously passing patches. This caused dramatic leaderboard reshuffling, with the previous top-ranked agent dropping from 78.80% to 62.20% and falling to fifth place. The work highlights the critical need for more rigorous evaluation of AI coding assistants beyond simple test-passing metrics.