Sparse Regression under Correlation and Weak Signals: A Reproducible Benchmark of Classical and Bayesian Methods

Hao Xiao's reproducible benchmark pits six sparse regression methods—OLS, Ridge, Lasso, Elastic Net, Horseshoe, and Spike-and-Slab—against each other under conditions that make sparse regression genuinely hard: correlated features (pairwise correlation up to 0.9), weak signals (four SNR levels), and growing dimensionality (p=20, 50, 100). The study runs over 2,600 experiments on synthetic data with three covariance structures plus the classic Diabetes dataset. The goal is to give practitioners a clear head-to-head comparison of classical penalized estimators (milliseconds to run) versus Bayesian priors (minutes via MCMC) for both prediction and variable selection.

The results are nuanced but decisive in key areas. Bayesian methods dominate prediction error: Horseshoe and Spike-and-Slab achieve a mean squared error (MSE) of 72, compared to classical methods ranging from 108 to 267. The Horseshoe also delivers near-nominal 95% coverage (94.8%), while Spike-and-Slab under-covers at 91.9%—likely due to its continuous relaxation. For variable selection, Lasso and Spike-and-Slab tie at an F1 score of ~0.47, making Lasso the practical default when full posteriors aren't needed. The takeaway: choose Bayesian for superior prediction and uncertainty quantification, but stick with Lasso for fast, no-frills feature selection.