Grokking as a Phase Transition between Competing Basins: a Singular Learning Theory Approach

A team of researchers including Ben Cullen and Sergio Estan-Ruiz has published a groundbreaking paper titled 'Grokking as a Phase Transition between Competing Basins: a Singular Learning Theory Approach' on arXiv. The work tackles the mysterious phenomenon of 'grokking,' where AI models trained on tasks like modular arithmetic show poor performance for extended periods before suddenly and dramatically improving in generalization. The researchers interpret this abrupt shift not as random luck but as a structured phase transition between competing 'solution basins' in the model's loss landscape. They achieve this by applying Singular Learning Theory (SLT), a Bayesian framework that analyzes the geometry of how models learn.

Their key contribution is using SLT's 'local learning coefficient' (LLC)—a measure of loss surface degeneracy—to mathematically track this transition. They derived closed-form expressions for the LLC in quadratic networks and provided empirical verification, showing that trajectories of this coefficient reliably signal when generalization is about to occur. This transforms grokking from an observed curiosity into a quantifiable, predictable process governed by the statistical properties of different solution basins. The findings provide a new diagnostic tool for researchers training models, potentially allowing them to predict when a stubborn model is on the verge of a generalization breakthrough, saving significant compute time and resources.