Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia

A team of researchers including Haining Pan, Nakul Aggarwal, and J.H. Pixley has published a groundbreaking paper that applies statistical mechanics to understand neural network pruning. By systematically varying dropout rates during both training and evaluation phases on the MNIST dataset, they discovered that pruning doesn't just gradually degrade performance but induces sharp phase transitions between three distinct states: 'eumentia' where networks successfully learn, 'dementia' where they've forgotten, and 'amentia' where learning is impossible from the start.

The study reveals that the transition between the functional 'eumentia' phase and the degraded 'dementia' phase exhibits scale invariance with a diverging length scale, showing hallmarks of a Berezinskii-Kosterlitz-Thouless-like transition familiar from condensed matter physics. This phase structure proved robust across different network architectures, maintaining consistency regardless of width or depth variations. The researchers demonstrated that the algebraic decay of loss in the eumentia phase—what machine learning practitioners recognize as neural scaling laws—corresponds to quasi-long-range order from a statistical mechanics perspective.

This work establishes dropout-induced pruning as a concrete experimental setting where neural network behavior can be rigorously analyzed through statistical physics frameworks. The identification of these distinct phases provides a new theoretical foundation for understanding why some pruned networks maintain performance while others catastrophically fail, potentially leading to more principled approaches to neural network compression and optimization in practical AI systems.