LLMs improve code via iterative feedback, reasoning models excel

A new study from Le Zhang and Suresh Kothari, presented at the 14th Computing Conference 2026, systematically investigates how Large Language Models (LLMs) can correct their own code using iterative feedback loops. Unlike most benchmarks that measure single-attempt accuracy, this work focuses on the real-world programming process of refinement. The researchers built a framework where LLMs receive compiler error messages and test case feedback after each attempt, then try again. They tested four models across two major programming languages, introducing novel metrics to evaluate code failures and rectification patterns.

The results show a clear advantage for reasoning models: they consistently improve over iterations, substantially outperforming non-reasoning models in leveraging feedback. Notably, syntactic and runtime errors are far more tractable than logical or algorithmic failures. The paper offers actionable insights for building better LLM-based code generation systems, emphasizing that iterative feedback loops are key to unlocking stronger performance. This work has direct implications for developers using AI coding assistants, suggesting that future tools should incorporate structured feedback mechanisms rather than relying on one-shot generation.