ReqElicitGym: An Evaluation Environment for Interview Competence in Conversational Requirements Elicitation

A research team from Peking University and collaborating institutions has introduced ReqElicitGym, a groundbreaking evaluation environment that systematically measures how well LLMs can interview users to uncover software requirements. The system addresses a critical gap in AI-assisted software development: while LLMs excel at generating code, their ability to ask the right questions and uncover what users actually need remains largely unquantified.

ReqElicitGym features 101 carefully constructed website requirements scenarios spanning 10 application types, from e-commerce to social media platforms. The environment includes both an interactive oracle user (simulating real users) and a task evaluator, both achieving high agreement with human judgments. This allows for reproducible, quantitative testing of any conversational requirements elicitation approach without relying on subjective human scoring or limited real-world interactions.

When researchers tested seven representative LLMs using their new benchmark, the results were revealing: current models uncover less than half (under 50%) of users' implicit requirements. The study found that effective questioning typically emerges late in conversations, and while LLMs can handle interaction and content requirements, they consistently struggle with style-related aspects. This systematic evaluation reveals that interview competence, not coding ability, is becoming the new bottleneck in LLM-based software development.

The implications are significant for both AI researchers and software engineering professionals. As automated development tools become more prevalent, understanding and improving how AI systems gather requirements will be crucial. ReqElicitGym provides the first standardized way to measure this capability, potentially guiding development of better prompting strategies, specialized models, or hybrid human-AI approaches for requirements engineering.