Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

A team of researchers including Jie Huang, Bruno Loureiro, and Stefano Sarao Mannelli has published a significant paper providing a sharp, interpretable description of the loss landscape for two-layer ReLU neural networks. In a teacher-student setting with Gaussian data, they demonstrate that local minima can be exactly represented by a small set of summary statistics. This breakthrough offers a clear mathematical picture of the optimization terrain, moving beyond vague intuitions. Furthermore, they establish a direct link to one-pass Stochastic Gradient Descent (SGD), showing these minima act as attractive fixed points in the dynamics of these summary statistics.

This analysis reveals a crucial hierarchical structure in the landscape. In a well-specified regime, minima are isolated. However, as the network becomes overparameterized (i.e., its width increases), these minima become connected by flat, low-loss directions. This connectivity fundamentally changes the optimization process, making global minima increasingly accessible and pulling the SGD dynamics toward them, thereby reducing the risk of getting stuck in poor local solutions. The findings challenge common simplifying assumptions in theoretical machine learning, showing that even minimal neural network models possess complex landscape features that are essential for understanding modern training success.