New study evaluates LLM-based automated bug repair efficiency
GPT-5 fixes 39 more complex bugs than DeepSeek-V3.2 but costs more
Researchers from Colorado State University and industry collaborators published an empirical analysis of LLM-driven bug fixing in *arXiv:2608.14065*, comparing two APR systems (ChatRepair and CodeCorrector) across three models: GPT-5, DeepSeek-V4-pro, and DeepSeek-V3.2. The study introduces a multi-dimensional framework to assess how bug complexity, fault localization accuracy, and model reasoning settings affect repair performance and cost-efficiency.
The team found that while structurally complex bugs and imprecise fault localization degrade repair success, low-cost models can still fix over 50% of moderately complex bugs. GPT-5 repaired 7 and 39 more complex bugs than DeepSeek-V4-pro and DeepSeek-V3.2 respectively, but incurred significantly higher computational costs. Surprisingly, higher-cost models and stronger reasoning settings did not consistently deliver better cost-efficiency, revealing a critical trade-off between repair effectiveness and expense.
- GPT-5 repairs 39 more complex bugs than DeepSeek-V3.2 but costs substantially more
- Fault localization precision has a major impact—imprecise localization widens the performance gap between APR techniques
- Low-cost models achieved competitive results on moderately complex bugs, challenging the assumption that bigger models always perform better
Why It Matters
This challenges the 'bigger model = better' paradigm in software engineering automation and highlights cost-sensitive deployment strategies for AI-driven bug fixing.