New arXiv study: Coding agent tool choice boosts speed 41.6%

A new arXiv paper from Xiangzhe Xu and eight co-authors (including Marc-Alexandre Côté from Microsoft Research) takes a systematic look at how tool architecture—the way capabilities are organized and exposed to a large language model—shapes coding agent behavior. While most prior work focuses on expanding what tools can do, this study controls for capability and varies only the interface. Across three different LLM actors and 11,700 trajectories on repository-level issue fixing, they compared six architectures: from a bare bash tool, to more structured low-level command interfaces, to natural-language search, to Python CodeAct-style execution, plus lightweight text-based cognitive scaffolds.

The results are striking. Moving from bash-only to structured low-level interfaces improved consistency across repeated attempts by up to 4.7x, implying that giving the agent a more defined API surface reduces random variance in how it explores the repo. Adding natural-language search broadened repository exploration and increased access to relevant files by more than 11%, helping agents find necessary code faster. But the biggest efficiency win came from Python CodeAct-style interfaces, which let the agent write and execute Python snippets directly: they achieved the same task success rate while using 41.6% fewer steps and 56.3% lower token consumption.

Meanwhile, lightweight text-based cognitive-scaffolding tools—like notes or intermediate reasoning records—had limited effect on agent behavior, suggesting that simple prompt-level memory aids don't move the needle when the underlying interaction loop is unchanged. The authors argue this is a critical but understudied design dimension, and their controlled benchmark offers a framework for thinking about "how" agent capabilities are exposed, not just "what" capabilities exist.

For developers and platform builders, this is a practical signal: swapping a bash-only interface for a CodeAct-style Python tool can cut agent operating costs by more than half, and structured tool APIs can dramatically improve reliability on repetitive tasks. As AI coding agents move from demos to production, the interface they're handed may matter as much as the model behind them.