New paper challenges depth benefits in norm-constrained neural networks

A new paper by Julia Nakhleh and Robert Nowak presents a unified function space theory for deep fully connected neural networks, defining functions recursively as ℓ¹-bounded linear combinations of activated functions from prior layers. This framework works for both homogeneous activations (like ReLU) and non-homogeneous ones, bridging gaps between norm-based complexity bounds and variational depth characterizations. The authors prove a novel representer theorem and function-space complexity bounds showing that these function classes remain small at any depth.

Most strikingly, for univariate ReLU networks, they prove a 'depth saturation' result: depth yields only a small constant rescaling of the function class, with no added functional diversity. Consequently, deep norm-controlled ReLU networks in any dimension cannot exhibit high frequencies along any direction. This reveals that commonly cited expressivity benefits of depth vanish once complexity is controlled by an appropriate function space norm rather than by parameter count or other representational costs that permit compounded rescaling across layers.