LLMs Plateau at 90.8% on VerilogEval — and Alignment Is Making the Problem Worse

A new paper from researchers including Guan-Ting Liu, Chao-Han Huck Yang, and others (affiliations include NVIDIA and academia) investigates how large language models fail and generalize in register-transfer level (RTL) coding — the parallel temporal logic used in hardware design. The authors introduce a novel error taxonomy grounded in problem solvability and cognitive theory, dividing failures into syntactic, semantic, solvable functional, and unsolvable functional types. By evaluating frontier models on the widely used VerilogEval benchmark, they uncover a strict empirical ceiling: even the best LLMs plateau at a 90.8% initial pass rate. This ceiling is driven by unsolvable functional errors, which represent knowledge gaps that resist test-time compute scaling.

The study also reveals a troubling “surface convergence gap”: optimization techniques easily eliminate syntax errors but simultaneously exacerbate deeper functional failures. The authors demonstrate that current alignment methods, such as reinforcement learning from human feedback, only teach models to produce syntactically valid code — not to truly understand hardware semantics. Repeated sampling can patch solvable errors, but the core capacity for RTL coding remains strictly bounded by the knowledge present in the pretraining data. The paper concludes that advancing LLM-based hardware generation requires focusing on model reasoning rather than alignment interventions, a finding with significant implications for automated hardware design pipelines.