ELM Network paper proves complex neurons beat simple ones under budget constraints

A new paper from Spieler, Martius, and Levina (arXiv:2605.12049) questions the long-held default in machine learning of using extremely simple neural units. The team introduces the ELM Network, whose recurrent layer is built from Expressive Leaky Memory (ELM) neurons — designed to mirror functional components of cortical neurons. This architecture allows independent adjustment of three key parameters: number of units N, per-unit effective complexity k_e, and per-unit connectivity k_c, and trains stably across orders of magnitude in scale. Evaluating on two qualitatively different sequence benchmarks — the neuromorphic SHD-Adding task and Enwik8 character-level language modeling — the researchers found that performance improves monotonically along each of the three axes individually when the others are held fixed. However, under a fixed total parameter budget P, a clear non-trivial optimum emerges in their tradeoff: larger budgets favor both more and more complex neurons. The paper proposes a closed-form information-theoretic model that explains these tradeoffs, attributing diminishing returns at the two ends to per-neuron signal-to-noise saturation and across-neuron redundancy.

A hyperparameter sweep spanning three orders of magnitude in trainable parameters traces a near-Pareto-frontier scaling law consistent with the framework. This suggests that the simple-unit default in machine learning is not obviously optimal once this tradeoff surface is probed. The work also offers a normative lens on why the cortex relies on complex spatio-temporal integrators: evolution under stringent biological constraints may have naturally settled on an optimal allocation that maximizes computation per parameter. With 25 pages, 21 figures, and 3 tables, the paper provides both empirical and theoretical grounding for a shift in how recurrent architectures are designed. For practitioners, this implies that investing in more expressive neurons could yield better performance per parameter than simply adding more simple units — especially at larger scales.