Embedding of Low-Dimensional Sensory Dynamics in Recurrent Networks: Implications for the Geometry of Neural Representation

A new theoretical neuroscience paper provides a fundamental explanation for a common observation in both biological and artificial neural networks: why their activity organizes into simple, low-dimensional geometric shapes, known as manifolds. Researchers Vikas O'Reilly-Shah and Alessandro Selvitella modeled cortical circuits as recurrent neural networks (RNNs) driven by predictable sensory inputs, like repeating patterns. They combined concepts from dynamical systems theory—generalized synchronization and delay embedding—to prove that a contracting RNN will almost always develop a smooth internal structure that mirrors the external sensory dynamics. Crucially, the requirement is modest: a network with just slightly more neurons than twice the intrinsic dimension of the data (N > 2d) is sufficient, aligning with classic mathematical bounds.

The work goes further by establishing a critical link between a network's function and its internal geometry. The authors proved a 'prediction-separation' principle: for a network to make accurate predictions about future sensory inputs, its internal states must be sufficiently separated from one another. The maximum allowable confusion between states is bounded by the prediction error. This mathematically explains phenomena like categorical perception (sharp boundaries between concepts) and discrimination thresholds. In numerical experiments, they trained tanh RNNs on circular data and observed the predicted ring-shaped manifolds emerge in the network's hidden states, with state separation improving sharply at the N=2d+1 threshold.

Interestingly, while their theory starts with the condition of a 'contracting' network, their experiments show that training pushes networks beyond strict contraction, yet the low-dimensional embeddings persist. This indicates their mathematical conditions are sufficient but not strictly necessary, offering a robust mechanistic account for why these geometric structures are so prevalent. The findings bridge abstract dynamical systems theory with concrete AI training, providing a new lens to understand and potentially design more efficient and interpretable recurrent models.