New Method Slashes the Cost of Testing AI Coding Helpers by Half

AI coding assistants — the tools that write and fix software for you — get updated constantly. Until now, checking whether each update actually helped meant running the assistant through hundreds of test tasks, a process that can cost thousands of dollars and days of waiting. A new open-source method called DeltaSelect, described in a paper by Nicholas J. Conn, offers a cheaper option: pick a small, fixed set of tasks that reliably predicts the overall result, then repeat just those.

Testing AI coding agents is noisy. Run the same AI on the same task twice and you often get different answers. The paper found that only about one in five test tasks (22 out of 113) gave results that reliably matched the full suite. In other words, most of that expensive testing was telling developers very little.

DeltaSelect uses statistics to find the tasks that do track the full picture, then converts partial scores onto one common scale so results stay comparable. Its goal isn't ranking which AI is best — it's answering a simpler question: did my latest tweak make things better or worse? In one case study, a developer used it to rewrite custom instructions. Across 13 rounds of testing — total cost $27.86 — the final version ran 58% cheaper ($1.75 versus $4.18 per run) and scored higher.

For anyone who pays for AI tools, the takeaway is straightforward: cheaper, faster testing means companies can improve their AI coding helpers more often instead of shipping changes blindly, and AI budgets stretch further. The catch: this method only compares your own tweaks, not one AI model against another, and someone technical still has to set it up.