New FSD metric predicts grokking 1,722 steps before transformers generalize
Fourier synchronization predicts AI's sudden learning leap with 99% precision across 9 tests.
A new paper by Achyuthan Sivasankar, published on arXiv (2606.12966), tackles the mysterious phenomenon of grokking—where transformers suddenly transition from near-random to perfect accuracy on tasks like modular arithmetic. The author introduces the Frequency Synchronization Degree (FSD), a normalized, permutation-tested metric that measures Fourier circuit synchronization without requiring prior circuit knowledge. Across nine setups (primes 53, 71, 97, 113, 131, three seeds), FSD synchronizes 500–3,000 steps before grokking, with a mean lead of +1,722 steps, outperforming existing baselines like excluded loss.
The study provides direct causal evidence that the inter-phase gap is driven by weight decay regularization. By forking training at the FSD-ceiling step and varying weight decay λ, grokking timing shifts monotonically earlier, with Δt ∝ 1/λ, fitting R²=1.00 and R²=0.99 for two clean cases across primes 53, 97, and 131. Architecture ablations reveal that attention-only models grok with a strong FSD precursor, while MLP-only models never grok, confirming that the phenomenon is a multi-block circuit property. This work opens new avenues for controlling and understanding emergent generalization in AI.
- FSD predicts grokking a mean of 1,722 steps before generalization, across 9 configurations (primes 53–131).
- Causal evidence: grokking delay Δt follows Δt ∝ 1/λ (weight decay) with R²=1.00 in clean cases.
- Attention-only models grok with FSD precursor; MLP-only models never do, proving multi-block circuit dependence.
Why It Matters
Enables early prediction and causal control of sudden generalization in transformers, key for reliable AI training.