Study: Coding Agents Need Minimal Context to Edit Code Effectively
Most context is wasted; only the code being edited matters for agent performance.
A new paper by Brian Sam-Bodden rigorously tests what context coding agents actually need when editing code. Using SWE-bench Verified, the study separates the “finding” phase from the “acting” phase by holding localization fixed with an oracle. The core finding: only the code being edited matters. Natural-language summaries of the target code answered just 4 out of 45 behavioral questions compared to 27/45 with the raw source—and a frontier model’s summaries performed identically to a 3B model’s, proving the gap lies in representation, not summarization quality.
The surrounding context is equally irrelevant. Across every multi-file instance, rendering a file’s remainder as UML skeletons and signatures resolved no more issues than deleting it outright (p=0.75). However, compressed context matches whole files at one-third the tokens—19K tokens per resolved issue vs. 94K—offering significant efficiency gains. The study also uncovers a critical noise floor: temperature-0 API inference flips ~9% of per-instance outcomes between byte-identical runs, meaning small effects reported on this benchmark may be unreliable. The released instrument includes gold-validated environments and pre-registered hypotheses for reproducibility.
- Natural-language summaries are ineffective: frontier model summaries scored as poorly as a 3B model's (4/45 vs 27/45 on held-out repos).
- Surrounding context doesn't matter: rendering file remainder as UML skeletons resolved no more issues than deleting it (p=0.75).
- Compressed context achieves same results with 1/3 tokens: 19K vs 94K tokens per resolved issue.
Why It Matters
Coding agents can slash token costs and latency by focusing only on the code being edited.