New Study: The Tools Around AI Coders Matter Less Than the Model
The expensive brand-name setup may not write better code — and it can cost more.
A new study asked whether AI coding tools work better when paired with their own maker's harness — the tools, prompts and control flow that turn a chat model into an autonomous software engineer. On average, no. Researchers ran paired same-model contrasts on a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks, with the same 80 tasks run under claude-agent-sdk and deepagents on claude-opus-4-8, and under the openai-codex SDK and deepagents on gpt-5.5. 792 of 800 planned runs were graded by an isolated oracle. Neither contrast resolved an average advantage: -1.25 percentage points for Opus 4.8 (48.8% vs 50.0%) and +1.25 pp for GPT-5.5 (55.6% vs 54.4%). The Opus average combines opposite strata — the native harness trails by 9.0 pp on the 61 repository tasks and leads by 23.7 pp on the 19 contest tasks — a partition chosen after seeing the data that needs a designed replication. Re-priced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8 and 1.2 times on GPT-5.5, but these are observed-usage estimates: 58 runs on the Anthropic account left no usage record, which could move the Opus ratio between 0.7 and 2.3, so the billed ordering is unresolved.
- Running the same AI model inside a competitor's tool made almost no difference to how many coding tasks it solved — about one percentage point either way.
- The brand-neutral setup cost 1.3 to 1.6 times more per solved task on Anthropic's model, though missing billing data means that number could flip.
- More than a quarter of runs stopped at the time limit had already produced a working fix, suggesting time limits — not skill — decide some failures.
Why It Matters
If you or your company pay for AI coding tools, brand-matched bundles may not buy better results — but they can cost more.