New Study Reveals How Model Width and Data Shape Neural Network Scaling Laws

A team of researchers including Julius Girardin, Emanuele Troiani, and Lenka Zdeborová from EPFL has published a theoretical study on arXiv that systematically explores how model width and training data jointly influence generalization in neural networks. Using a simplified quadratic two-layer network with ℓ2 regularization, they derived explicit expressions for the test error as a function of the number of trainable parameters (width) and the number of samples. Their analysis goes beyond classic fixed-feature or infinite-width regimes to capture the effect of feature learning in a finite-sample setting with structured data.

The results reveal a rich phase diagram with multiple scaling regimes. The generalization error follows power laws that depend sensitively on the spectral structure (eigenvalue decay) of the target function. As width increases, the model transitions between regimes, including an interpolation threshold where training error vanishes but test error may still improve. The work provides a theoretical framework for understanding when adding parameters helps or hurts generalization, and how optimal model size depends on data quantity and regularity. This opens the door to more principled architecture design in practice.