PerfCodeBench tests LLMs on high-performance code optimization

A team of researchers from multiple institutions introduced PerfCodeBench, a new executable benchmark designed to evaluate large language models (LLMs) on system-level, high-performance code optimization. Unlike existing benchmarks that focus on functional correctness or algorithmic problem-solving, PerfCodeBench targets realistic systems-level tasks requiring hardware-aware implementation choices, careful management of performance bottlenecks, and parallelism or GPU operations. Each task includes executable correctness checks, a baseline implementation, and a reference optimized solution, enabling evaluation of both correctness and runtime efficiency.

Evaluating a broad set of state-of-the-art LLMs, the researchers found a significant gap between model-generated code and expert-optimized implementations. The gap was especially pronounced for tasks involving parallelism and GPU usage. Current models also demonstrated weaknesses in cross-language robustness and consistently failed to reach expert-level efficiency. The results underscore the need for performance-aware evaluation to push LLMs beyond generating merely correct code toward producing efficient systems software. The benchmark data, evaluation infrastructure, and complete logs are publicly available.