Pythia training law reveals why LLM capabilities snap into place

A new preprint from Lei Dong (arXiv:2607.27281) reframes how language models acquire skills, treating capability formation as a rare-event nucleation process — the AI equivalent of a physical phase transition. The key idea: for a capability to exist, all parts of its underlying circuit must align in one stochastic attempt. Getting all but one right is worthless, so the waiting time for that final alignment depends directly on how many parts are missing, not on the model's total size. Dong validates this with a 'shortcut-free' experiment where a five-part circuit missing three parts waits as long as a three-part circuit missing three parts (ratio 1.19-1.37), proving the wait counts missing parts, not model scale. Extending to the Pythia suite across seven capabilities and three scales, ablating a single circuit part leaves a median 17% of the capability in 32 of 32 discriminating cells — while a random non-part head leaves 100%. This decisively rejects partial-credit models, which would predict 50-83% (p = 2e-10).

The resulting rate law — sites × attempts × drive × exp(-β*K) minus destruction — is then read in both directions. Forward: capabilities that are flat at baseline ignite exactly at a chosen training step once a concentration floor is crossed (10/10 successes above, 0/12 below), and their arrival time can be predicted from precursor signals to 5% median error on six held-out models. Backward: withholding a capability delays its learning, but past a critical training step it never ignites at all — even as validation loss falls smoothly, leaving standard monitoring blind. The damage is localized to heads committing to base data, and the cure is surprisingly specific: re-initializing only the query-key slices of attention heads restores learnability in 6/6 cases, while value slices do nothing (0/6). Dong also proves the mechanism in a controlled gated-attention model, and shows SGD's noise fails the fluctuation-dissipation test, prompting a custom anneal to melt and pin circuits on schedule. For practitioners, this suggests capability emergence is controllable and plasticity loss is reversible — if you know which slices to reset.