New Study Proves Weight Norm Directly Controls Neural Network Grokking Timing

Grokking—the phenomenon where a neural network suddenly generalizes long after overfitting training data—has puzzled researchers. Some studies suggested weight norm triggers the transition, but others observed grokking without a critical norm. A new paper from Truong Xuan Khanh and colleagues cuts through the debate by directly intervening on weight norm during training, rather than just observing it. Under free training with weight decay, networks grok when the norm reaches a value Wc that varies little across seeds and learning rates (CV 1–2%) and grows as a power law with modular base. When they clamped the norm to a fixed multiple ρ of Wc, grokking still occurred, but the delay T_grok followed an exponential law: T_grok ∝ exp(α ρ), with α≈7.5 and R²=0.996 across four moduli. The held norm shifted delay by about 19×, while learning rate only shifted it 2×. Holding norm above Wc slowed grokking but did not prevent it.

Critically, the researchers found that adding a LayerNorm after training removed the dependence entirely by decoupling weight scale from network function—without it, the exponential law returns. This pinned-norm delay is the exponential counterpart to the logarithmic delay predicted for a freely contracting norm. The paper provides the first causal evidence that weight norm is the underlying mechanism setting the grokking timescale. For machine learning practitioners, this means that controlling weight norm could allow predictable timing of generalization in deep neural networks, with implications for training schedules, regularization strategies, and understanding how models transition from memorization to generalization.