New theory explains neural networks' tendency towards simple solutions
Critical points in deep networks force parameters to simpler subnetworks, new proof shows.
Researchers from multiple institutions have published a paper on arXiv that deepens our understanding of why neural networks naturally converge to simpler solutions—a phenomenon often linked to Occam's Razor. The work, titled "Singular Learning and Occam's Razor in Deep Monomial Networks," focuses on fully-connected networks with monomial activation functions. By applying Mason's Theorem—a result from polynomial algebra used to analyze function field extensions—the authors characterize critical points where the Jacobian of the parametrization becomes rank-deficient. These singularities are central to Singular Learning Theory, which studies how model architecture impacts optimization dynamics.
The key finding is that for sufficiently high activation degrees, critical points correspond precisely to subnetworks formed by deactivating or redundantly combining neurons. This means gradient-based optimization is mathematically biased toward finding parameters that collapse into simpler substructures. The result provides a rigorous explanation for why deep networks often ignore unnecessary complexity during training. While the current result holds for monomial activations (like x^d), it suggests a broader principle: architectural singularities naturally enforce simplicity. This could influence future designs of activation functions and regularization techniques to encourage generalization.
- Uses Mason's Theorem from polynomial algebra to analyze critical points in deep monomial networks
- Shows that for large activation degree, criticality occurs exactly at subnetworks with inactive or redundant neurons
- Provides a mathematical foundation for implicit bias toward simpler functions in deep learning
Why It Matters
This mathematical grounding could inspire new architectures and training methods that naturally favor generalization.