Benchmarking Requirement-to-Architecture Generation with Hybrid Evaluation

A research team led by Minxiao Li has introduced R2ABench (Requirement-To-Architecture Benchmark), a first-of-its-kind dataset designed to evaluate how Large Language Models (LLMs) perform at generating software architecture designs from Product Requirements Documents (PRDs). The benchmark pairs real-world software projects with expert-curated PlantUML reference diagrams, providing a standardized foundation for testing. To assess the outputs, the team developed a multi-dimensional, hybrid evaluation framework that goes beyond simple accuracy checks. This framework analyzes generated diagrams across three complementary layers: Structural Graph Metrics, Multi-dimensional Scoring, and Architecture Anti-pattern Detection.

Using R2ABench, the researchers conducted a comprehensive empirical study evaluating state-of-the-art models and agentic workflows. The results reveal a critical gap in current AI capabilities. While LLMs like GPT-4 and Claude 3 demonstrate strong syntactic validity and robust entity extraction—correctly identifying components—they fundamentally struggle with relational reasoning. This leads to structurally fragmented and incoherent architectures where the connections and dependencies between components are poorly defined. Interestingly, code-specialized models partially alleviated this limitation, but agent frameworks, often touted for complex tasks, introduced significant instability rather than delivering consistent improvements. The study concludes that R2ABench provides the necessary tool to advance LLM-driven software architecture generation beyond its current, syntax-heavy limitations.