New paper proves deep ReLU nets can match kernel methods with optimal generalization
First minimax-optimal rates for deep networks trained with standard GD and SGD.
A team of five researchers (Zhou, Wang, Lei, Ying, Zhou) has published a landmark theoretical paper on arXiv, establishing the first known minimax-optimal generalization rates for deep ReLU networks trained with either gradient descent (GD) or stochastic gradient descent (SGD). Prior work had shown optimal rates only for shallow networks within the neural tangent kernel (NTK) regime, leaving a glaring gap for deep architectures. The new analysis proves that, under the condition that network width grows polynomially with both depth and training sample size, the excess population risk of GD and SGD converges at the optimal rate—matching the best possible performance achievable by any kernel method.
This result has significant implications for the theory of deep learning. It suggests that deep neural networks can be as statistically efficient as classical kernel methods when trained with simple gradient-based optimizers, provided the width is scaled appropriately. The paper also provides the first unified treatment of GD and SGD for deep ReLU networks, showing both can attain the same optimal rates. While the analysis is theoretical, it offers a rigorous foundation for why deep networks generalize well in practice, potentially guiding architecture design and training recipes for real-world applications.
- Establishes minimax-optimal generalization rates for deep ReLU networks under GD and SGD for the first time.
- Requires network width to scale polynomially with depth and sample size (e.g., width ~ depth^a * sample^b).
- Matches the optimal rates of kernel methods, bridging the theory gap between shallow and deep architectures.
Why It Matters
Proves deep networks can be as statistically efficient as kernel methods, providing a theoretical foundation for generalization in practice.