New SPDNN estimator tackles covariate shift in deep regression with optimal rates

Covariate shift—where training and test data come from different distributions—frequently breaks standard machine learning models. In a new arXiv paper, Kengne and Ockegna introduce a sparse-penalized deep neural network (SPDNN) that tackles this problem head-on, specifically for nonparametric quantile and Huber regression. The approach is designed to work with dependent data (e.g., time series) as well as i.i.d. observations. When the density ratio between source and target distributions is unknown, the method incorporates a two-step pre-training procedure: first estimating the ratio via a least squares SPDNN, then using that estimate to reweight the regression estimator. The authors provide non-asymptotic error bounds for the SPDNN estimators in the class of Hölder smooth functions.

Crucially, the proposed estimators are adaptive—they can achieve minimax optimal convergence rates (up to a logarithmic factor) not only for i.i.d. data but also for several classical dependent data models, including φ-mixing, strong mixing, and C-mixing processes. This is enabled by a generalized Bernstein-type inequality satisfied by many such processes. The work unifies and extends previous results on deep regression under covariate shift by handling dependent data and multiple loss functions (quantile and Huber) simultaneously. For practitioners, this offers a principled way to build robust regression models when training and deployment environments differ, without sacrificing statistical efficiency.