New Math Predicts How Long AI Training Takes to 'Warm Up'

Deep inside an AI model is a giant grid of numbers, called weights, that get adjusted during training. As training goes well, those numbers settle into a distinctive shape: most are tiny, but a small number become very large. Researchers treat that lopsided pattern — nicknamed a "heavy tail" — as a sign of healthy learning. Until now, nobody had a good answer for a basic question: how long does it take to show up?

This paper offers one. The author ran 330 controlled training experiments and found that the waiting time doesn't just depend on how quickly the model's early learning takes off. Instead, it grows with the model's size, following a tidy mathematical rule. One smart tweak: runs that never reached the pattern before time ran out weren't thrown away. They were counted as "not yet," much like counting runners still out on a marathon course rather than pretending they never started.

The work also checks real, publicly available models — Qwen2.5-0.5B and Pythia-70M — and finds their number patterns don't look like random noise. Popular training methods Adam and AdamW behaved the same way; simpler methods didn't reach the pattern at all in the same conditions. The author is careful to say this doesn't prove Adam causes the pattern, only that the timing is predictable.

Why care? Training AI is enormously expensive, burning electricity and money for days or weeks. Knowing roughly when a run will "settle" could help teams plan, spot trouble early, and avoid wasting computing. But keep expectations in check: this is a single-author preprint, tested on small models and textbook setups, not on today's giant systems. It's a useful building block, not a finished tool — and it won't change anything you use tomorrow.