Learning reveals invisible structure in low-rank RNNs

Yoav Ger and Omri Barak’s latest paper, submitted to arXiv on May 5, 2026, tackles a fundamental gap in our understanding of learning in recurrent neural networks (RNNs). They extend the popular low-rank RNN framework from static connectivity analysis to dynamic learning by deriving closed-form ordinary differential equations (ODEs) that govern gradient descent directly in a reduced overlap space. For linear RNNs the derivation is exact; for nonlinear RNNs it becomes asymptotically exact in the large-N Gaussian limit. This mathematical tool lets them distinguish two classes of hidden structure: loss-visible overlaps that fully determine network activity and output, and loss-invisible overlaps that have no effect on function yet are essential for capturing the learning trajectory.

The practical implications are significant. First, the authors demonstrate that learning itself can serve as a perturbation that exposes hidden differences in connectivity between networks that are otherwise functionally equivalent—a potential diagnostic for comparing trained models. Second, loss-invisible overlaps can act as memory variables that encode the entire training history, long after the network’s output has converged. The paper characterizes exactly when this memory persists. Finally, the theory yields several testable predictions for biological learning experiments, bridging the gap between theoretical ML and neuroscience. With 30 pages and 12 figures, this work offers a new mathematical lens for understanding how neural representations evolve during training.