Agentic LLMs hit 84% pass rate on Terraform generation via iterative refinement

A new arXiv paper presents a verifier-first empirical study of seven agentic strategies for generating Terraform Infrastructure-as-Code (IaC) from natural language. Using the IaC-Eval v2 benchmark (186 AWS/Terraform tasks with Rego v1 policies), the authors break down failures into three verifier stages: validate, plan, and opa eval. The study applies statistical rigor with McNemar's test and Wilson confidence intervals.

Key findings include: (1) Active retrieval via ReAct agents with MCP or ChromaDB-backed RAG lifts Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1 (p<0.0001). (2) Iterative refinement using verifier feedback achieves 62.9% (Qwen 7B) and 84.4% (GPT-4o) pass@1, with binary convergence — tasks either resolve in one retry or exhaust the budget. (3) GEPA reflective prompt optimization adds +7.5 percentage points (p=0.026) using only 80 verifier-guided rollouts. (4) SIMBA teacher-free demonstration injection matches Active RAG performance without retrieval infrastructure but fails on SELF_DEFINED_PROPERTY errors. (5) A diagnostic injection experiment shows 79% of post-refinement OPA failures are information-gap errors resolvable when policy text is visible, motivating policy-aware generation.