Blind Resampling beats self-repair in small code models by up to 6 points

A new paper from Yuvraj Verma challenges the conventional wisdom that code agents should use self-repair—showing a failed program to the model along with test output and asking for a correction. Using a placebo-controlled design on the MBPP+ benchmark across model scales of 1.5B, 3B, and 7B parameters, the study compares four retry strategies: blind resampling (regenerating from scratch), a generic failure notice, genuine execution feedback, and feedback plus verbal self-reflection. The results are striking: blind resampling emerges as the strongest condition below 7B and remains statistically tied with the best at 7B, while consuming 2.5–5.5× fewer tokens. Execution feedback adds no measurable benefit over a placebo failure notice, and conditioning on the model's own failed attempt actually reduces performance by 6.1 points at the 1.5B scale (p=0.006). The culprit is anchoring—when shown its previous attempt, a model reproduces a near-identical program in 33–68% of retries, compared to just 2–14% under blind resampling.

Two follow-up experiments reinforce the finding. First, injecting retrieved solutions to other tasks has no significant effect (bounded to ±3.5 points), localizing the harm to self-conditioning rather than context length. Second, verbal reflection is the only condition that measurably weakens the anchor but remains dominated on cost—reflection still underperforms blind resampling when controlling for token budget. The paper replicates the results across two model families and two precision levels (full vs. reduced), ruling out competing explanations like precision or architecture differences. Across six configurations, the magnitude of the anchoring penalty is predicted solely by baseline code quality (r=0.96). In essence, the cost of committing to a bad first attempt outweighs any potential benefit from feedback, especially in smaller models. For practitioners building code agents, this suggests that simply retrying from scratch is more efficient and effective than complex self-repair pipelines.