Study: Giving LLMs a spec checklist boosts code correctness by 38%
A single prompt line change makes LLM-generated code 38% more reliable across models.
A new study by Amin Haeri and Mahdi Ghelichi on arXiv (2607.06636) isolates the key factor that makes LLMs generate correct code: grounding tests in the specification. The researchers held the tester, test budget, and repair loop fixed, only changing a single prompt line that controls whether the tester receives the spec as a checklist of rules. Against a strong baseline—already instructed to probe invalid inputs and edge cases—adding spec grounding produced correct code +38 percentage points more often across three Claude tiers (Haiku 4.5, Sonnet 4.6, Opus 4.8), +28 points on GPT-5.3-codex, and +19 points on Gemini 3.5 Flash.
Importantly, the improvement comes from the spec's content, not its format or sheer test quantity. Doubling the test budget barely helped, and combining eight independent ungrounded test suites plateaued far below grounding. In an ablation, giving the spec as a plain paragraph recovered 27 of 30 bugs, but asking for tests without the spec recovered only 2. Property-based generators caught 28 of 30 bugs but invented out-of-spec requirements. Grounding also slashed false alarms from 33% to 0% against a Python standard-library oracle. The implication: a simple prompt tweak can dramatically boost LLM code reliability without extra computational cost.
- Grounding tests in the specification increased correct code by +38 percentage points on Claude models vs. a strong baseline.
- Doubling the test budget had negligible effect; combining eight ungrounded test suites plateaued far below spec-grounded results.
- False alarm rate dropped from 33% to 0% when tests were grounded in the spec, reducing wasted debugging effort.
Why It Matters
A simple prompt change can dramatically improve LLM code reliability, reducing debugging time and false positives.