New math makes shallow neural networks trainable in one step

A new paper on arXiv (arXiv:2607.02003) introduces a paradigm shift for training shallow neural networks. Instead of grappling with non-convex loss landscapes and stochastic heuristics, the authors – Matej Benko, Pierre Bousquet, Iwona Chlebicka, and Błażej Miasojedow – replace the discrete training problem with a continuum variational surrogate. They identify a family of λ-convex functionals over parameter densities in weighted Sobolev spaces, proving the variational problem is globally well-posed, stable, and exhibits unexpected almost C³ regularity. Unlike existing Wasserstein-based or mean-field approaches that face discretization challenges, this formulation provides direct access to elliptic regularity and convex analysis.

The key breakthrough: the optimal parameter density can be obtained by solving a single linear system, completely eliminating the need for iterative gradient-based optimization. The authors also establish explicit generalization error controls at a rate of 1/α relative to the regularization parameter, and prove that finite-width networks of size N achieve the continuum optimum at an O(1/N) rate. This theoretical framework bridges the gap between the Neural Tangent Kernel (NTK) and feature-learning regimes, offering a principled understanding of over-parameterization through variational calculus.