RDNN network masters continuous working memory with divisive normalization

A team of researchers has introduced the Recurrent Divisive Normalization Network (RDNN), a new neural architecture that overcomes a long-standing failure of standard RNNs in continuous working memory tasks. Traditional continuous attractor networks require precise fine-tuning to hold variables like angles or positions, while GRUs and LSTMs typically collapse continuous representations into discrete point attractors. RDNN borrows divisive normalization—a computation common in cortical circuits—to create algebraic, activity-dependent division that lets the network learn high-fidelity slow manifolds without hand-tuned parameters.

Through dynamical systems analysis and Backpropagation Through Time, the team showed that divisive normalization induces a local gradient scaling that dampens parameter updates in highly active regimes. This leads to a self-compression of effective rank, confining dynamics to a low-dimensional subspace while avoiding explicit low-rank factorization pathologies. Ablations revealed that subtractive inhibition maintains static memories but fails under time-varying inputs; divisive normalization is mathematically necessary to prevent manifold shattering. These insights position divisive normalization as a critical computational mechanism, not just a biological artifact, with implications for building robust AI memory systems.