ReLU networks: Depth beats width for analytic functions in new arXiv paper

A new paper from researchers Yanming Lai, Defeng Sun, and Yang Wang (arXiv:2607.10589) tackles a fundamental question in neural network approximation theory: how do depth and width trade off when approximating functions with infinite smoothness (analytic functions)? Most prior work characterized approximation rates using a single parameter like total parameters. However, a 2020 paper by Shen et al. introduced a dual-parameter (N,L) approach, where N is width and L is depth. For function classes with finite smoothness s, that work established a symmetric rate of O(N^{-2s/d} L^{-2s/d}), meaning depth and width contribute equally.

This new paper flips that intuition for analytic functions. The authors derive a superior rate of O(N^{-C L^τ}) where C>0 and τ>0 depends on the relation between L and N. Crucially, τ=1 when N scales roughly as L^d. This exponential improvement in depth demonstrates that depth is inherently more powerful for representing smooth, analytic functions. The technical challenge involved carefully constructing ReLU networks to approximate power functions, multivariate multiplication, and polynomials—building blocks that may prove useful for other approximation tasks. The paper spans 47 pages with 4 figures and 1 table, and is submitted to arXiv's Machine Learning (stat.ML), Information Theory, Numerical Analysis, and related categories.