New Study: AI Learns Better From Fewer, Fresher Practice Questions

Training an AI is a bit like coaching a student for a math contest. You give the model practice problems (researchers call these "prompts"), let it attempt answers, and correct it. One popular method has a smarter AI act as the tutor, grading and guiding a smaller AI. The obvious assumption has always been: more practice problems is better. This team decided to test that assumption directly.

They fixed everything else — the same 14,080 practice attempts, the same 110 rounds of correction — and only varied two things: how many different questions the AI saw, and how often its practice answers were swapped out for fresh ones from its newer self. The results flipped depending on that second choice. When answers were frozen at the start, going from a few questions to thousands dropped accuracy from 21.16% to 19.05%. When answers were refreshed each round, the same increase pushed accuracy up, from 23.61% to 25.57%. With fresh answers, a mere eight questions got 24.09% — nearly matching the full 14,080-question result of 24.51%.

A second test found another trade-off. Models trained with periodic refreshes did better on short, quick answers and finished more of them. But given a much larger answer budget, the frozen-answer models eventually caught up in average accuracy — while burning 1.7 to 1.8 times as many words to get there. So "better" depends on how you're measuring, and on how much you're willing to spend.

The catch: these are small accuracy numbers, roughly a quarter correct on tricky math, and this is a lab study, not a finished product. Nobody should expect an overnight leap in the chatbot on their phone. But the direction is genuinely useful — it suggests the huge expense of training AI may come down by being smarter about freshness rather than simply buying more data.