The Inductive Bias of Convolutional Neural Networks: Locality and Weight Sharing Reshape Implicit Regularization

A team of researchers including Tongtong Liang, Esha Singh, Rahul Parhi, Alexander Cloninger, and Yu-Xiang Wang has published a groundbreaking paper titled 'The Inductive Bias of Convolutional Neural Networks: Locality and Weight Sharing Reshape Implicit Regularization' on arXiv. The research provides the first mathematical proof explaining why convolutional neural networks (CNNs) generalize so much better than fully connected networks on tasks like image recognition. While previous work established that fully connected networks suffer from insufficient regularization on difficult distributions like high-dimensional spheres, this paper demonstrates that CNN architectures fundamentally change this dynamic through their built-in inductive biases of locality and weight sharing.

The researchers proved that when the receptive field size (m) remains small relative to the ambient dimension (d), CNNs can generalize on spherical data with a rate of n^(-1/6 + O(m/d))—a regime where fully connected networks provably fail. This occurs because weight sharing couples learned filters to the low-dimensional patch manifold, effectively bypassing the high dimensionality of the ambient space. The team further validated their theory by analyzing natural image patch geometry, showing standard convolutional designs induce patch distributions that align perfectly with this stability mechanism. This work provides a systematic, mathematically rigorous explanation for the empirical success of CNNs that has been observed but not fully understood since their breakthrough performance on ImageNet in 2012.