Local cycles and interneurons key to RNN computational power

A new paper by Tom Talpir and Elad Schneidman from the Weizmann Institute of Science tackles a fundamental question: how does the wiring of a neural network determine what it can compute? The researchers built and tested thousands of recurrent neural networks (RNNs) with varying architectures, training each to solve a large set of Boolean functions. For small networks, they constructed complete 'catalogs' of network-function performance, revealing that computational capacity varies widely—most networks are poor at most tasks, and most functions are hard to compute.

The key insight: networks containing local 2- and 3-cycle motifs (short loops in the connectivity graph) show dramatically enhanced performance. These cyclic architectures are often the minimal designs that can solve specific functions. The team also found that a small set of structural statistics can accurately predict a network's performance. Extending to larger networks, they discovered that typical networks fail even to approximate randomly chosen functions. However, adding a small number of sparsely connected, biologically inspired interneurons—similar to inhibitory neurons in the brain—dramatically increases computational capacity. This effect is distinct from simply adding more nodes or connections; the presence of short cycles remains the critical factor. The work offers a general framework for understanding structure-function relationships in computing networks.