Yang & Sun's online sparse regression hits optimal rates via overparametrization
No parameter tuning needed—a new hard-thresholding algorithm solves online sparse regression with closed-form updates.
Sparse regression is a cornerstone of modern machine learning, but most work has focused on offline settings where the full dataset is available. Yang and Sun's new paper, "Online Generalized Sparse Regression: How Does Overparametrization Help?" tackles the online formulation head-on, addressing four key challenges: dynamic regularization parameter updates, storage/memory complexity, real-time computation, and optimal statistical guarantees.
The authors introduce an online hard-thresholding algorithm that operates on a constrained formulation—eliminating the need to tune regularization parameters across rounds. The algorithm performs closed-form updates and only requires storing summary statistics, making it computationally, memory, and storage efficient. Despite the nonconvex, combinatorial nature of the problem, the method achieves global convergence at the optimal statistical rate under realistic assumptions, provided the projection set is properly overparameterized. Numerical experiments on cardinality-constrained linear regression and low-rank matrix sensing show consistent improvements over state-of-the-art baselines, suggesting a practical edge for streaming data applications.
- Proposes online generalized-sparsity-constrained regression for linear regression and low-rank matrix sensing, avoiding dynamic regularization tuning.
- New online hard-thresholding algorithm uses closed-form updates and stores only summary statistics, cutting memory and compute overhead.
- Achieves global convergence at optimal statistical rates with proper overparametrization; outperforms state-of-the-art in numerical experiments.
Why It Matters
Enables real-time sparse regression on streaming data with no tuning—critical for online learning, signal processing, and large-scale recommendation systems.