Feature Bagging Boosts ML Stability, New arXiv Paper Shows

A new paper from Yuheng Ma and Qiang Sun, posted on arXiv (2607.26964), introduces a framework for understanding why feature bagging—the ensemble technique of training models on random subsets of features—improves algorithmic stability. The authors propose 'feature instability' (FI), a feature-axis analogue of the well-known instance instability (II). While II measures how model output changes when a single data point is removed, FI captures sensitivity to removing a single feature. Experiments demonstrate that FI provides generalization-relevant information not captured by II alone.

The paper provides formal stability guarantees for feature bagging in both parametric linear models and model-free settings inspired by recursive feature subsampling in random forests. Key results show that feature bagging consistently improves stability compared to non-bagged counterparts, with larger improvements under more aggressive subsampling. Importantly, the authors prove that a modest number of bagging rounds is sufficient to approach the stability level of infinite bagging, making the technique practical for real-world ML pipelines. This theoretical grounding offers a new lens for designing stable, generalizable models.