SWE-Pro benchmark shows LLMs lag behind 15.5x expert speedups
LLMs achieve negligible speedups vs. experts' 15.5x on real-world optimization tasks
A new paper on arXiv introduces SWE-Pro, a repository-level benchmark derived from 102 expert-performed optimizations across open-source projects. Unlike prior benchmarks that oversimplify code optimization to isolated functions, SWE-Pro evaluates runtime, peak memory, and a composite Time-Weighted Memory Usage (TWMU) metric under noise-aware conditions. The benchmark tests LLMs on parameterized real-world inputs and execution variability, aiming to reflect actual engineering challenges.
Results show LLMs struggle dramatically: runtime improvements are negligible, and memory optimizations are virtually nonexistent. In contrast, expert implementations yielded an aggregate 15.5x speedup and 171.3x peak memory reduction. Experts improved runtime in 91.2% of tasks and peak memory in 65.7%. The findings underscore a fundamental shortfall in current LLM code reasoning, suggesting that models lack the deep understanding needed for nontrivial performance engineering.
- SWE-Pro benchmark uses 102 expert-written optimizations from real open-source repositories with noise-aware measurement.
- Current LLMs achieve negligible runtime gains and virtually no memory optimization on these tasks.
- Expert code achieved 15.5x speedup and 171.3x peak memory reduction, succeeding in 91.2% and 65.7% of tasks respectively.
Why It Matters
Reveals a critical gap: LLMs can't yet handle real-world performance optimization, limiting their use in production engineering.