Spectral Edge Dynamics Reveal Functional Modes of Learning

A new research paper by Yongzhong Xu introduces 'spectral edge dynamics' as a novel framework for understanding how neural networks learn. The study reveals that during the phenomenon of 'grokking'—where models suddenly generalize after prolonged training—learning concentrates along a small number of dominant update directions. These directions, termed the spectral edge, reliably distinguish grokking from non-grokking regimes. Crucially, standard mechanistic interpretability tools like head attribution, activation probing, and sparse autoencoders fail to capture these directions because their structure isn't localized in parameter or feature space.

Instead, each spectral edge direction induces a structured function over the input domain, revealing low-dimensional functional modes invisible to representation-level analysis. For modular addition, all leading directions collapse to a single Fourier mode. For multiplication, the same collapse appears only in the discrete-log basis, yielding a 5.9x improvement in concentration. For more complex tasks like computing x²+y², no single harmonic basis suffices, but cross-terms of additive and multiplicative features provide a 4x variance boost. The research demonstrates that training discovers functional modes whose structure depends on the algebraic symmetry of the task, with multitask training amplifying compositional structure and increasing concentration by 2.3x.

These findings suggest that spectral edge dynamics identify the fundamental functional subspaces governing learning, whose representation depends on the task's algebraic structure. Simple harmonic structure emerges only when tasks admit symmetry-adapted bases, while more complex tasks require richer functional descriptions. This represents a significant advancement beyond current interpretability approaches that focus on parameter or feature space analysis.