Double Descent Found in Least-Squares Interpolation on Contaminated Data

A new simulation study by Tino Werner explores whether the double descent phenomenon—where generalization error decreases after a model complexity threshold—occurs in linear regression when training data is contaminated. In robust statistics, contamination refers to data points that deviate from the assumed ideal distribution, often acting as outliers that can severely distort classical estimators like least-squares interpolation. Werner compares the performance of this highly non-robust estimator with several robust alternatives across varying levels of model overparameterization.

The results reveal that large overparameterization indeed triggers a double descent for the least-squares interpolator: after an initial peak in test error, the error drops sharply as the model becomes more overparameterized, eventually outperforming robust estimators. This suggests that extreme model complexity can compensate for the lack of robustness, allowing even a flawed estimator to generalize well on corrupted data. The finding has implications for both theory and practice, as it challenges the need for robust methods when models are sufficiently overparameterized.