ReqGenX achieves 96-99% alignment decomposing legacy SRS documents
New pipeline turns outdated requirements into traceable LLM training data with near-perfect fidelity.
Evaluating automated Software Requirements Specification (SRS) generation has long been hampered by a lack of datasets with fine-grained traceability between source requirements, intermediate artifacts, and generated specs. To address this, Ragib Shahariar Ayon and colleagues propose ReqGenX, a controlled pipeline that transforms legacy SRS documents into synthetic pre-SRS artifacts. The pipeline first decomposes SRS sections into atomic statements grounded in the original source, then routes these atoms to standards-inspired artifact types using multi-LLM plurality voting. Finally, it generates artifacts with constrained prompts and iterative judge-guided refinement. The approach was tested on seven PURE SRS documents, measuring grounding, quality, information retention, and downstream reconstruction.
The results are striking: atoms produced by ReqGenX show median AlignScore values typically between 0.96 and 0.99, and Prometheus scores ranging from 4.34 to 4.85, indicating high faithfulness and usability. Generated artifacts remain strongly grounded in source atoms, with AlignScore values between 0.80 and 0.94 and near-100% judge pass rates under standard evaluation. However, stricter Prometheus evaluation reveals pass rates from 54.8% to 97.1%, highlighting trade-offs between faithfulness and completeness. In a downstream SRS reconstruction case study, artifact-backed atoms remained recoverable with SBERT means between 0.69 and 0.75 and AlignScore medians between 0.76 and 0.84. The paper, accepted at ESEM 2026, concludes that traceable synthetic pre-SRS artifacts can support more fine-grained evaluation of LLM-based SRS generation while exposing key trade-offs.
- Atomic decomposition achieves median AlignScore of 0.96–0.99 and Prometheus scores of 4.34–4.85 across 7 PURE SRS documents.
- Stricter Prometheus evaluation yields artifact pass rates ranging from 54.8% to 97.1%, showing trade-offs between faithfulness and completeness.
- Downstream SRS reconstruction retains alignment with AlignScore medians of 0.76–0.84, enabling traceable evaluation.
Why It Matters
Enables fine-grained evaluation of LLM-based requirements generation, a key step toward reliable automated software engineering.