Adaptive Norm-Based Regularization for Neural Networks

In a new arXiv preprint (arXiv:2605.00171), Muhammad Qasim and Farrukh Javed tackle a core challenge in neural network training: how to regularize effectively when features are correlated or high-dimensional. Standard approaches like weight decay (ℓ2) and lasso (ℓ1) treat all features independently, ignoring covariance structure. The authors introduce two penalty variants: Covariance-Aware Ridge (CAR), which modifies the ℓ2 penalty by weighting each parameter according to the covariance of its corresponding input feature, and Sparse Covariance-Aware Ridge (SCAR), which adds an ℓ1 sparsity term on top of the covariance-weighted ℓ2 penalty. This allows the network to shrink or zero out irrelevant features while preserving structure among correlated inputs.

Using Monte Carlo simulations and two real-world datasets—a building cooling-load prediction task and a leukemia cell-type classification from high-dimensional gene expression data—the proposed methods consistently outperform standard weight decay and lasso in terms of out-of-sample predictive performance and model complexity control. The improvements are most pronounced when features are highly correlated or the dimensionality is large relative to sample size. The paper provides theoretical intuition and extensive empirical validation, suggesting that practitioners using neural networks on structured data with correlated features should consider these adaptive regularization schemes as drop-in replacements for standard weight decay.