ISO-grounded NFR specs improve LLM code quality, not correctness

A new study from João Pedro Monteiro Pereira and Vinicius Cardoso Garcia (accepted at SBCARS 2026) asks whether grounding non-functional requirements in the ISO/IEC 25010 quality model improves LLM code generation. The researchers compared three prompt styles on HumanEval and HumanEval-ET: a terse one-line baseline (NL-simple, RobuNFR-style), rich natural-language prose grounded in ISO 25010 (NL-rich), and structured JSON with the same ISO content. They tested four NFRs—performance, error handling, code smell, and readability—using ten prompt variations per condition under a fixed model snapshot, with paired non-parametric statistical analysis.

The primary finding is nuanced. ISO-grounded enrichment significantly improves static quality proxies: unreadability density dropped across all four NFRs, with performance falling from 0.88 to 0.69 under the NL-rich condition. It also reduces sensitivity to prompt wording, making outputs more consistent. However, these quality gains did not translate into better functional correctness. In fact, for error handling, the extended-test pass rate decreased, suggesting that defensive coding patterns learned from richer NFR descriptions conflict with exact-output benchmarks. A secondary finding: when the underlying ISO content was held constant, NL-rich and structured JSON approaches differed negligibly in correctness (|delta| ≤ 0.023), indicating semantic content is far more important than serialization format. The authors recommend practitioners invest in standard-grounded NFR content rather than debating JSON-vs-prose, and they provide a fully traceable replication package.