Grokking explained: noise-driven escape from metastable phases

A new paper from Ibrahim Talha Ersoy and Karoline Wiesner (arXiv:2606.17120) provides a mechanistic explanation for grokking—the sudden, delayed onset of generalization after prolonged overfitting in deep neural networks. By analyzing linear DNNs under varying L2 regularization strengths, the authors demonstrate that grokking corresponds to hysteresis in first-order phase transitions. When regularization is tuned to deliberately trap networks in low-accuracy metastable states, escape occurs only when SGD noise provides enough energy to cross a barrier, with escape times scaling according to the Arrhenius law. They reproduce grokking-like behavior across two orders of magnitude in escape time and match canonical curves where test error eventually catches up to training error.

The findings suggest that the number of metastable states equals the number of learnable features (one per singular value of the data covariance), meaning hysteresis grows naturally with task complexity. Importantly, the authors provide evidence that the same mechanism likely applies to general nonlinear DNNs. This reframes grokking from an anomaly to an emergent property of optimization landscapes shaped by regularization. The work opens routes toward more efficient learning schemes by predicting and controlling the timing of generalization, potentially enabling faster training or deliberate use of metastable phases for better final performance.