New benchmark shows LLMs flunk cloud infrastructure code edits

Building infrastructure-as-code (IaC) is critical for modern cloud systems, yet LLM capabilities in this domain remain underexplored. Existing benchmarks focus on declarative tools like Terraform and full code generation, not the incremental edits common in enterprise development. SWE-InfraBench fills this gap by sourcing tasks from dozens of real-world AWS CDK repositories, requiring models to modify existing code based on natural language instructions and pass test cases.

Results reveal significant limitations: Sonnet 3.7 (Anthropic) tops the leaderboard at just 34% success, while specialized reasoning models like DeepSeek R1 achieve only 24%. This stark underperformance highlights that current LLMs struggle with cloud resource dependencies and implementation patterns, even as they excel in general code generation. The benchmark, accepted at the NeurIPS 2025 Workshop, provides a standardized way to track progress.

The findings have immediate implications for DevOps teams considering AI-assisted IaC. Current models are far from reliable for production cloud infrastructure, where errors can cause outages or security vulnerabilities. As enterprises increasingly adopt IaC, improving LLM performance on tasks like these becomes essential. The SWE-InfraBench dataset is available for researchers to benchmark new models.