Study: Citing Requirements in LLM Code Enables Hallucination Detection but Reduces Determinism

A new empirical study from arXiv investigates the trade-off between output determinism and verifiability in LLM-generated code under Spec-Driven Development (SDD). The author, Subham Panda, tested three frameworks: traceSDD (mandatory per-line REQ identifiers), Spec Kit (artifact-level user stories), and OpenSpec (post-hoc trace maps). Experiments involved two frontier models—Claude Sonnet 4.6 (240 implementations across 4 conditions) and GLM-5-turbo (600 implementations)—measuring lexical similarity and automated hallucination detection rates (TDR). The pre-registered analysis reveals a consistent cross-model trade-off: the uncited condition produces significantly higher determinism (Claude: d=-0.76, p=0.003; GLM: d=-0.72, p<0.001), while only the cited condition enables any automated hallucination detection (TDR: Claude 86.4%, GLM 88.0%, vs 0% for all alternatives, with 0% false positive rate).

Notably, traceSDD (cited) significantly outperforms Spec Kit on determinism (Claude: d=0.47, p=0.049; GLM: d=0.42, p=0.003) but not OpenSpec (Claude: d=0.18, p=0.44; GLM: d=0.14, p=0.32). This suggests that finer-grained citation annotations improve determinism over coarser artifact-level traces, but still fall short of the consistency achieved without any citations. The findings establish that citation annotations inherently trade determinism for verifiability, and this trade-off generalizes across model architectures. For practitioners, this means choosing between predictable code generation and the ability to automatically detect hallucinations—a decision that depends on whether reliability or auditability is the higher priority.