SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

A team of researchers from UW-Madison and other institutions has published SlopCodeBench, a groundbreaking benchmark that exposes a critical weakness in current AI coding assistants like GPT-4 and Claude 3. Unlike traditional benchmarks that test single-shot solutions, SlopCodeBench evaluates how coding agents perform across long-horizon, iterative tasks where they must repeatedly extend their own code under evolving specifications. The results are stark: across 11 tested models, no agent solved any of the 20 problems end-to-end, with the highest checkpoint solve rate reaching only 17.2%. The benchmark tracks two key quality metrics—verbosity (redundant code) and structural erosion (complexity concentration)—revealing that agent code degrades steadily with each iteration.

When compared against 48 open-source Python repositories, agent-generated code was found to be 2.2 times more verbose and significantly more structurally eroded than human-written code. Tracking 20 repositories over time showed human code quality remained stable, while agent code deteriorated with each extension. The study also conducted prompt-intervention experiments, finding that while initial code quality could be improved, degradation continued unabated. These findings demonstrate that current pass-rate benchmarks systematically underestimate the challenges of iterative software development, and that today's AI agents lack the architectural foresight and design discipline required for real-world engineering workflows where code must be maintained and extended over time.