LLMs turn business requirements into test assertions on 10 real bugs
Five LLMs generated Java oracles directly from natural-language requirements without code.
Deep Dive
Researchers evaluated five LLMs (DeepSeek-V3, Gemma-3n, Llama-3, Mistral-7B, Qwen-3) on generating test oracles from natural-language business requirements for 10 real bugs in Defects4J Lang. They found non-trivial generalization but substantial variance across bugs and models. Generated oracles aligned more closely with the requirement-derived gold standard than with the system under test.
Key Points
- Tested 5 LLMs on 10 real bugs from Defects4J Lang using requirement-derived oracles as gold standard.
- LLM oracles aligned more with requirements (REQ) than with actual system behavior (SUT).
- No detectable linear relationship between requirement ambiguity and oracle accuracy suggests pretraining coverage dominates correctness.
Why It Matters
LLMs could automate test creation from business specs, but high variance means human oversight remains essential for reliability.