Maximum Entropy Networks Reveal How Neural Structure Balances Randomness and Task Constraints

A fundamental question in neuroscience is how network connectivity relates to function. Traditional methods train networks via gradient descent on cognitive tasks and then analyze the resulting weights, but the structure depends heavily on details like initialization and learning rate. In a new preprint, Ludwig Hruza and Srdjan Ostojic from ENS Paris offer a complementary perspective using the maximum entropy principle. They model connectivity as a probability distribution over single-neuron weights, with task requirements expressed as constraints on that distribution. By maximizing Shannon entropy subject to those constraints, they derive the most random network that still solves the task. A tunable weight scale parameter controls the balance between randomness and task-induced structure, making the approach independent of any particular learning algorithm.

Applying this framework to context-dependent input-selection tasks in 2-layer feed-forward networks, the authors show that maximum entropy inference becomes analytically tractable by mapping nonlinear networks onto gain-modulated linear models. Starting from an a priori homogeneous distribution, maximizing entropy under task constraints leads to the emergence of distinct populations of neurons, each defined by its pattern of contextual gain modulation. Increasing the number of contexts drives a transition from context-specialized to unspecialized, random populations. Increasing the weight scale drives a parallel transition from structured to random stimulus selectivity. Strikingly, the resulting maximum entropy connectivity matches both qualitatively and quantitatively the structure of networks trained with gradient descent across different learning regimes. These results suggest that the interplay between task constraints and entropy maximization provides a fundamental principle for understanding the relationship between structure and function in neural networks.