Towards Supporting Quality Architecture Evaluation with LLM Tools

A team of computer science researchers from Spanish universities has demonstrated that large language models can outperform human experts in evaluating software architecture quality. Their paper, "Towards Supporting Quality Architecture Evaluation with LLM Tools," documents a study where Microsoft Copilot was tested against experienced software architects using the Architecture Tradeoff Analysis Method (ATAM), a rigorous framework for assessing design tradeoffs between competing quality attributes like performance, security, and maintainability.

In the experiment, students generated quality-attribute scenarios which were then analyzed by both human architects and the Copilot LLM. The AI tool not only matched but often exceeded human performance in identifying architectural risks, sensitivity points, and tradeoffs. The researchers found Copilot produced "better and more accurate results" in most cases while dramatically reducing the time and effort required for what is traditionally a manual, brainstorming-intensive process.

The findings suggest that generative AI has significant potential to transform software engineering workflows by automating parts of the architecture evaluation pipeline. Rather than replacing architects, tools like Copilot could serve as intelligent assistants that suggest more qualitative scenarios for evaluation and recommend the most suitable design approaches for specific contexts. This represents a concrete step toward AI-augmented software development where LLMs handle routine analytical tasks, freeing human experts for higher-level strategic decisions.

The research team—Rafael Capilla, Jorge Andrés Díaz-Pace, Yamid Ramírez, Jennifer Pérez, and Vanessa Rodríguez-Horcajo—argues this approach could make architecture assessment more efficient and accessible, particularly for complex systems where balancing competing quality requirements is challenging. Their work, available on arXiv, provides empirical evidence that LLM tools are ready to move from coding assistants to architectural advisors.