Linear equivalence of nonlinear recurrent neural networks

A new theoretical paper on arXiv by David G. Clark tackles a fundamental problem in the study of large recurrent neural networks (RNNs): understanding the structure of their high-dimensional activity. When RNNs have random couplings, they can generate complex, potentially chaotic patterns of neural firing. The key object encoding this collective behavior is the N×N covariance matrix of neural activity. Prior analytical work could only compute low-dimensional summary statistics, not the full matrix for a specific set of couplings. Clark's paper provides a rigorous derivation showing that, for large N, the covariance matrix of a nonlinear RNN is equivalent to that of a linear network with the same couplings, driven by independent noise. This 'linear equivalence' was first proposed as an ansatz, but Clark now proves it using two complementary versions of the two-site cavity method, a technique from statistical physics.

The first derivation decomposes each unit's activity into a linear component and a nonlinear residual, demonstrating that cross-covariances between residuals are strongly suppressed—so residuals act as independent noise. The second derivation writes a self-consistent matrix equation for the covariance matrix, carefully separating Gaussian and non-Gaussian contributions that enter at the same order. A naive Gaussian closure would give the wrong equation, but the cavity method produces the correct one. Numerical verification across a range of network sizes confirms the predictions. This work extends linear equivalence from feedforward systems, where weights are independent of inputs, to recurrent networks, where activities depend on the same couplings that generate them. The result provides a powerful analytical tool for understanding neural computation, learning dynamics, and emergent behaviors in complex systems.