InferenceBench tests AI agents' ability to optimize LLM inference speed

AI agents are increasingly used to automate research and development tasks, but existing benchmarks often test them on narrow, recipe-like workflows. InferenceBench, introduced by Yeon, Rank, and Andriushchenko, flips this by requiring agents to deploy an OpenAI-compatible inference server and optimize the speed of a given LLM on a single H100 GPU within a two-hour wall-clock budget. The benchmark isolates four distinct optimization scenarios: prefill latency, decode latency, concurrent request throughput, and a balanced mix of all three. Across 15 frontier agent configurations, agents reliably improved over a naive PyTorch baseline (up to 8.08x) and often matched or exceeded default serving engines like vLLM (4.05x), yet they still fell below a simple hyperparameter search under the same time budget (up to 11.53x).

Qualitative analysis of agent trajectories reveals a critical insight: although agents enumerate many relevant optimization techniques, they overwhelmingly converge on a single inference framework. They test only a few distinct configurations and spend the remaining budget re-measuring, repairing, or tuning hyperparameters rather than exploring substantially different strategies. This suggests the bottleneck is not domain knowledge but the ability to propose diverse configurations, evaluate them systematically, and submit the best solution found. InferenceBench marks a shift toward open-ended AI engineering benchmarks where memorized solutions yield limited improvements, pushing the field to focus on agent exploration and systematic decision-making.