Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations

A new theoretical breakthrough from researchers Yuhao Liu, Zilin Wang, Lei Wu, and Shaobo Zhang demonstrates that the choice of activation function fundamentally changes what neural networks can learn. Their paper, 'Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations,' proves that networks with smooth activation functions (like sigmoid or tanh) can automatically adapt to arbitrarily high orders of smoothness in target functions while maintaining constant depth, achieving minimax-optimal approximation and estimation error rates up to logarithmic factors.

Technically, the team characterized networks over Sobolev spaces W^{s,∞}([0,1]^d) for any smoothness s>0. Their constructive framework produces explicit neural network approximators with controlled parameter norms and model size, ensuring statistical learnability under empirical risk minimization without impractical sparsity constraints. The key finding: while ReLU networks' approximation order is strictly limited by depth—requiring proportional depth growth to capture higher-order smoothness—smooth activations provide this capability inherently.

This work challenges the dominance of ReLU in modern deep learning by identifying activation smoothness as a fundamental mechanism alternative to depth for attaining statistical optimality. The results suggest that for certain problem classes, shallow networks with carefully chosen smooth activations could match or exceed the performance of deeper ReLU architectures, potentially leading to more parameter-efficient models. The paper provides rigorous mathematical justification for reconsidering activation function selection beyond mere empirical performance, offering new theoretical tools for architecture design that prioritizes statistical efficiency over sheer depth.