Comprehensive Evaluation of Large Language Models on Software Engineering Tasks: A Multi-Task Benchmark
AI models that get the same perfect score can be 53 times more expensive to run.
Deep Dive
A new benchmark tested 11 leading AI models on five core software engineering tasks: bug fixing, feature development, code refactoring, technical writing, and research synthesis. While coding tasks saw 100% success, research tasks were harder. Crucially, models with identical perfect scores showed massive variation—up to 53x in cost and 22x in completion time. Tool usage did not predict success. The study highlights major efficiency gaps despite similar quality outputs.
Why It Matters
This reveals that choosing an AI for development requires evaluating both performance and operational cost, not just capability.