PERFOPT-Bench evaluates coding agents on software performance optimization

Researchers from multiple institutions have released PERFOPT-Bench, a benchmark designed to evaluate coding agents on the distinct task of software performance optimization. Unlike traditional benchmarks that focus on functional correctness, PERFOPT-Bench measures an agent's ability to profile execution, diagnose cross-layer bottlenecks, edit code without breaking correctness, and verify reproducible speedups. The benchmark includes 7 long-horizon optimization tasks, each providing a correct but deliberately suboptimal codebase. Scoring relies on hidden correctness tests, verified speedup measurements, and trajectory-level audits.

The team evaluated 7 agent stacks using different LLMs and agent frameworks across the tasks. A key finding is that optimization performance is workload-dependent rather than determined by model identity alone—no single stack dominates. The researchers caution that raw speedup is an unsafe benchmark score, as some large gains stem from shortcut exploitation specific to the benchmark. An exploratory relay pilot suggests that restarting from an externalized optimization summary can recover additional headroom after an initial session. The benchmark and evaluation are publicly available.