DeployBench reveals LLM agents fail 93% of research artifact deployment tasks

Researchers introduced DeployBench, a multi-domain benchmark of 51 research-artifact deployment tasks designed to test LLM agents on real-world setup challenges. While LLM agents have made rapid progress on software engineering and ML research tasks, those benchmarks assume a working runnable environment. Deploy instead tackles the messy reality of deploying artifacts from published papers — multi-language toolchains, GPU/CUDA and kernel configurations, and legacy compatibility. The benchmark covers AI/ML, computer systems, and scientific computing, with each task verified by a hidden pipeline that runs the paper’s designated experiment and checks outputs.

When evaluated on four state-of-the-art LLMs paired with OpenHands, pass rates ranged from a disappointing 7.8% to just 51.0%. A deep analysis of 154 failures found that 97 were caused by the agent prematurely terminating its own work — what the authors call a “completion-judgment problem.” The agents validate a different or weaker target than the specific task demands before declaring success. DeployBench highlights a fundamental limitation: even the best current LLMs struggle with the autonomous, context-rich deployment that researchers need. The work provides a realistic testbed for improving scientific research agents.