Efficient Approximation to Analytic and $L^p$ functions by Height-Augmented ReLU Networks

A team of researchers has published a groundbreaking paper on arXiv titled 'Efficient Approximation to Analytic and $L^p$ functions by Height-Augmented ReLU Networks.' The work, led by ZeYu Li, FengLei Fan, and TieYong Zeng, addresses two core limitations in neural network approximation theory. Their key innovation is a three-dimensional network architecture that enables a dramatically more efficient representation of sawtooth functions. These functions are a cornerstone for approximating complex mathematical function classes, meaning this breakthrough has ripple effects across the theoretical foundation of deep learning.

First, the team establishes substantially improved exponential approximation rates for several important classes of analytic functions, paired with a parameter-efficient network design. Second, and more significantly, they derive—for the first time—a quantitative and non-asymptotic approximation of high orders for general $L^p$ functions. This moves the field beyond qualitative statements to concrete, measurable bounds on approximation error. The techniques advance the theoretical understanding of how neural networks approximate functions in fundamental mathematical spaces.

The implications are profound for AI model design. By providing a 'theoretically grounded pathway,' this research points toward architectures that can achieve the same or better performance with far fewer parameters. This could lead to smaller, faster, and less computationally expensive models for tasks requiring high-precision function approximation, which underlies many scientific computing and engineering applications. It represents a major step in bridging the gap between deep learning practice and rigorous mathematical theory.