SWE-NFI benchmark exposes coding agents' weak spot: code quality
Best coding agents nail 70% functional tests but fail at structural improvements.
Coding agents are increasingly judged on functional correctness, but real-world engineering demands more: clean, maintainable, and well-structured code. The SWE-NFI benchmark, created by researchers from multiple universities including Pengyu Xue and Weiyi Shang, addresses this gap by evaluating agents on behavior-preserving non-functional improvements (NFIs). It comprises 188 tasks derived from actual merged pull requests in open-source Python projects, operationalized into 92 executable rules that cover maintainability, readability, and structural design.
When tested against state-of-the-art commercial and open-source agents, the results are stark. The best agent passes 70% of functional tests, yet all agents significantly trail human developers on NFI capability. The most dramatic shortfall appears in structural code improvements, where agents score between 0.0 and 1.3, compared with a human reference score of 1.5. This shows current models excel at "making it work" but struggle with "making it clean," a critical skill for production codebases. The benchmark provides a reproducible foundation for the AI coding community to push beyond correctness into quality.
- SWE-NFI benchmark includes 188 tasks from real merged Python pull requests, with 92 executable rules for non-functional improvements.
- Top coding agents achieve 70.0% functional correctness but only 0.0–1.3 on structural NFI scores vs. 1.5 for human developers.
- Benchmark covers behavior-preserving improvements like refactoring and readability, which are underexplored in current agent evaluations.
Why It Matters
Coding agents must do more than pass tests — maintainable code is production-ready code.