SmellBench reveals LLM agents struggle with architectural code smell repair

A new study by researchers from the University of Craiova and University Politehnica of Bucharest introduces SmellBench, the first framework designed to evaluate large language model agents on repairing architectural code smells – cross-module design flaws that erode software maintainability. Unlike localized bugs, these smells require understanding of overall system architecture, a challenge both for developers and automated tools. The team tested 11 agent configurations from four major model families—GPT, Claude, Gemini, and Mistral—on 65 hard-severity architectural smells detected by PyExamine in the Python project scikit-learn, with all results validated against expert judgments.

The findings reveal a sobering reality: 63.1% of the smells flagged by tools were false positives, and even the best-performing agent only resolved 47.7% of genuine smells. While agents could identify false positives with near-expert agreement (kappa = 0.94), their repair attempts often backfired – the most aggressive agent introduced 140 new smells. This inverse relationship between repair aggressiveness and net codebase quality exposes a fundamental gap: current LLMs excel at localized code transformations but lack the architectural reasoning needed for safe cross-module refactoring. SmellBench provides reusable infrastructure for tracking progress on this critical but underexplored dimension of automated software engineering.