New Study: AI That Looks Smarter May Just Be Lucky
The AI that wins on the first try may lose when you give it more tries.
Here's the setup. One of the main ways companies improve AI today is to reward it for answers that can be automatically checked — a math result, a piece of working code. Models trained this way usually beat their untrained originals when you ask once. But ask the same question fifty times and let the model pick its best shot, and the trained version sometimes falls behind. Researchers read this "crossover" as proof that training only sharpens skills the model already had, rather than teaching it anything new. This paper says that conclusion is shakier than it looks.
First, the statistics. The authors compared models on identical questions and built confidence bands — essentially margin-of-error bars, like in a political poll. Across five publicly available model pairs, not one crossover was statistically solid in the original tests. When they ran a much longer evaluation, allowing answers up to 32,000 tokens (roughly 25 pages of text), a genuine reversal showed up: the trained model started losing after somewhere between 11 and 61 attempts. Their key practical tip: adding more varied questions helps far more than letting the AI answer each question more times.
Second, they found the relationship isn't a single tidy curve. Two questions that were equally hard before training can behave completely differently afterward, and those differences repeat reliably. The right way to describe it is a distribution of outcomes per question, not one average line. Fitting that distribution correctly predicted crossovers in fresh runs and fixed the earlier model's error estimates. They also show how heavy losses on a handful of the hardest questions can erase an early lead, even while most questions improve.
So what does this mean for you? AI is sold to you — and to your boss — on the claim that each version is smarter. This work says some of those wins are within the range of chance, and that the way models are tested can hide real weaknesses. It doesn't prove today's AI is secretly bad, but it does mean benchmark headlines deserve a skeptical read before you bet a budget, a job, or a product on them.
- A common AI training method makes models better at first tries but sometimes worse when allowed many attempts — and that reversal usually isn't proven by the data.
- Testing five public model pairs, the researchers found no statistically solid crossover until they used much longer answers (32,000 tokens), where losses began after 11 to 61 attempts.
- More varied test questions are more useful than more attempts per question — a rule anyone evaluating AI should steal.
Why It Matters
It warns you to question 'smarter AI' claims before spending money, hiring, or trusting a product's marketing.