Here's a number that should make every AI team nervous: out of 113 coding benchmark tasks, only 22 carried real predictive signal. That's roughly 19.5%. The other 80%? Decorative. Expensive noise dressed up as science. This finding comes from a fresh arXiv paper introducing DeltaSelect — a method for cheaply checking whether your AI coding assistant is actually getting better. And it's a quiet bomb under the entire leaderboard economy.
Picture the typical AI dev team. They want to know if their new prompt tweak helped. So they run the full DeepSWE coding benchmark — hundreds of tasks, hours of compute, real dollars. Or they grab a handful of "representative" tasks and hope for the best. According to DeltaSelect's research, that second approach isn't a cheap approximation. It's closer to flipping a coin with a confidence interval.
DeltaSelect's fix: use repeated published DeepSWE trials to pick a small, frozen task set within a stated budget. Then run baseline-versus-candidate comparisons against that fixed set. In one case study using gpt-5.6-luna in a low-reasoning setting, 13 evaluations cost $27.86 total. That's the cost of a mediocre lunch. The setup also cut costs by 58% compared to the alternative — $1.75 versus $4.18 — while scoring higher. Cheaper and better signal. That's not a tradeoff. That's a mugging.
But here's the twist. DeltaSelect is explicitly not a leaderboard, not a universal model ranking, and not a full benchmark replacement. The authors repeat this disclaimer everywhere. Why? Because they see the trap. A frozen, cheap task set is cheap to tune against. The moment teams start publishing "DeltaSelect scores," the same Goodhart's Law dynamics that corrupted big benchmarks — where the metric becomes the target and stops measuring anything real — re-emerge in miniature. The authors aren't being modest. They're building a firewall.
So who wins? Small teams, indie devs, open-source maintainers. Anyone who could never justify a five-figure benchmark run. Also tooling vendors pushing the "your production harness isn't the benchmark harness" argument — like Playwright, now marketing token-efficient browser automation for coding agents. Who loses? Benchmark operators and leaderboard sites whose authority depends on expensive, centralized evaluation. And vendors who quietly optimized against known benchmark task sets? Their stable target just got blurry.
The biggest loser, though, is the assumption that your internal eval set is fine. That 22-of-113 stat implies a lot of teams have been shipping on vibes dressed up as measurements. You might be one of them.
Everything here rests on a single case study with one model. Whether those predictive 22 tasks transfer across models, reasoning levels, and harnesses is an open question. If they don't, the budget argument weakens. But the core insight — that arbitrary task selection is indefensible — feels durable. Trends suggest the next fight won't be about bigger benchmarks. It'll be about smarter, smaller, and honest ones. The question is whether the industry can resist turning DeltaSelect into just another leaderboard to game.