CLOSER-Bench: New benchmark exposes hardware AI agents' completion–closure gap

Hardware engineering is a long-horizon process where an AI agent must navigate delayed, heterogeneous feedback from tools like synthesis, static timing analysis, and place-and-route. Existing benchmarks measure individual stages—RTL generation, verification, PPA optimization—but do not capture how an agent recovers when a backend failure forces a revisitation of earlier RTL code. CLOSER-Bench (arXiv:2607.16632) closes this gap by pairing three task families (spec-to-RTL, RTL-to-GDS, and spec-to-GDS) for the same design and hidden objective, recording every tool invocation. It is built entirely on open-source infrastructure: Verilator for simulation, Yosys for synthesis, OpenROAD for physical design, KLayout for GDS, and Sky130 as the target technology.

A pilot of 10 tasks—covering RTL repair, mutation-based verification, coverage, PPA optimization, design-space exploration, cross-model debugging, and security—reveals a striking “completion–closure gap.” While three agents independently solve a localized AXI repair task, the matched verification-closure task separates a frontier agent from two otherwise successful baselines. The authors further validate a full RTL-to-GDS flow and construct a macro-based AXI/DMA streaming accelerator. The results indicate that hardware closure should be treated as a budgeted sequential decision problem, not a collection of independent code generation tasks, motivating new agent strategies that plan across abstraction boundaries.