New arXiv study reveals when overfitting turns benign or catastrophic

Zhijun Liu and Dandan Jiang's new paper, 'High-dimensional ridgeless least squares interpolation under spiked covariance structures,' tackles a core puzzle in machine learning: why overparameterized models that perfectly fit training data often still generalize well. Using a generalized spiked covariance model with multiple latent factors, the authors derive sharp asymptotic limits for out-of-sample prediction risk when the feature dimension p and sample size n grow at the same rate. The key innovation is showing that prediction behavior is governed by the alignment between the regression coefficient β and the spiked eigenspaces of the population covariance matrix—not just by the eigenvalue spectrum. Depending on how much signal energy concentrates along latent spike directions, interpolation leads to benign, tempered, or catastrophic overfitting.

The framework is notably general: spiked eigenvalues may be bounded or diverge at arbitrary rates, and their number can stay finite or grow with n. Crucially, the results hold under minimal moment conditions—only finite fourth moments are required, avoiding restrictive Gaussian assumptions common in similar analyses. The authors also characterize how the number, strength, and geometry of spikes jointly influence the double-descent curve, providing a unified theoretical lens for when latent covariance structures help or hurt generalization. For practitioners, this clarifies that in ridgeless regression (no explicit regularization), the spectral structure of features and the orientation of the true signal together determine whether overfitting is harmless or disastrous. The paper offers a rigorous foundation for understanding overparameterized models beyond standard i.i.d. assumptions, with direct implications for interpolating classifiers, random feature models, and neural networks in the lazy-training regime.