REAG framework cuts LLM testing costs by 31.7% with calibrated oracles
New ASE2026 framework hits 98.8% verdict accuracy while filtering unreliable AI outputs
An automated acceptance testing framework for LLM-based software introduces Requirements-Augmented Generation (REAG), which retrieves software requirements, domain knowledge, and user personas via adaptive RAG and self-reasoning to generate context-aware test oracles. A confidence-calibrated cascade judgment measures verdict reliability through simulated expert agreement—accepting high-confidence verdicts, escalating ambiguous cases, or abstaining when uncertain—with conformal risk control guarantees. In an industrial case study on a nutrition advisory application, REAG achieved a 3.91/5 oracle quality score, with qualified or marginal quality in 82% of cases. The cascade reached 98.8% accuracy, improved oracle quality to 4.30 by filtering unqualified outputs, and delivered a 31.7% cost-efficiency gain over single-judge baselines.
- REAG uses adaptive RAG + self-reasoning to generate context-aware test oracles from requirements, domain knowledge, and personas
- Confidence-calibrated cascade judgment hits 98.8% accuracy and lifts oracle quality from 3.91 to 4.30/5
- 31.7% cost-efficiency improvement over single-judge baselines, validated on a production nutrition advisory app (ASE2026)
Why It Matters
Gives LLM software teams a reliable, cost-effective way to automate acceptance testing and catch bad outputs before release.