New Theory Explains How RNNs Learn Long-Term Memory Through Eigenvalue Dynamics
Linear RNNs master memory via a single outlier eigenvalue – mathematicians prove it.
A new paper from Bordelon, Cotler, Pehlevan, and Zavatone-Veth (arXiv:2503.18754) tackles a fundamental question: how do recurrent neural networks learn to maintain memory over long timescales? By studying linear RNNs trained to integrate white noise, the researchers built an analytically tractable model of learning dynamics. They discovered that when initial recurrent weights are small, the entire learning process condenses into tracking a single outlier eigenvalue of the weight matrix. This eigenvalue directly corresponds to the long integration timescale, showing exactly how gradient descent sculpts the network's memory capacity.
The theory extends beyond simple integration to damped oscillatory filters, where learning involves a conjugate pair of outlier eigenvalues. The mathematical framework connects recurrent learning in machine learning to biological neural circuits, offering insights into how brains might acquire sustained activity patterns. This work provides a rigorous foundation for designing better recurrent architectures for tasks requiring long-range dependencies, from natural language processing to time series forecasting. It also opens doors for studying more complex nonlinear RNNs with similar analytical tools.
- Learning linear RNNs for integration reduces to tracking a single outlier eigenvalue of recurrent weights.
- The theory extends to damped oscillatory filters, involving a conjugate pair of outlier eigenvalues.
- Mathematical framework bridges machine learning and neuroscience by explaining how slow modes emerge via gradient descent.
Why It Matters
Provides rigorous mathematical insight into how neural networks learn memory, guiding better architectures for long-sequence tasks.