Study: Claude and DeepSeek models boost fix rate but not code quality
Newer AI models fix more bugs but generate code no better than older versions.
A new study from researchers at Linköping University questions whether newer AI coding models produce better quality code or simply fix more bugs. The team evaluated four models—two from Claude and two from DeepSeek—on the SWE-bench Lite repository-level benchmark, using the same SWE-agent functional repair pipeline. They measured non-functional quality via static analysis tools CodeQL and CodeScene, plus runtime metrics like CPU time and peak memory usage on the benchmark's test workload.
After statistical correction for multiple comparisons, the results were clear: later models resolved more instances overall, but on tasks solved by both generations, there was no significant improvement in any non-functional indicator. CodeQL rule violations were mostly identical, CodeScene category differences did not survive correction, CPU time varied inconsistently between model families, and peak memory was slightly higher for newer models. The authors conclude that the industry’s focus on resolved rates masks stagnation in code quality, urging a more holistic evaluation of AI's practical software engineering impact.
- Four Claude and DeepSeek models were compared on SWE-bench Lite with identical SWE-agent settings.
- After Holm correction, no static analysis (CodeQL, CodeScene) or runtime (CPU, memory) differences were statistically significant.
- Later models fixed more bugs overall but showed zero consistent non-functional quality improvement on common tasks.
Why It Matters
AI code generators are fixing more bugs but not improving code quality – quality metrics remain stagnant.