MM-XGBoost: Robust regression variant handles outliers without sacrificing accuracy
A two-step MM-estimator makes XGBoost robust to vertical outliers and leverage points
XGBoost is a go-to for tabular regression, but its standard quadratic loss can be thrown off by vertical outliers (extreme y-values) and leverage points (extreme x-values). In a new arXiv paper, Iris Aragón Mladosich and Christophe Croux systematically show how these data perturbations degrade XGBoost's predictions, even when using the built-in Huber loss. They explore alternative loss functions based on M-, S-, and τ-estimators from robust regression, and then combine them into a two-step procedure they call MM-XGBoost—the key finding of the study.
MM-XGBoost delivers the best trade-off between robustness and prediction accuracy, outperforming standard XGBoost and Huber-loss variants on contaminated datasets while staying competitive on clean data. The paper, spanning 30 pages plus 15 of supplementary material, benchmarks the method with 3 figures and provides a full theoretical grounding. For practitioners, this means a practical upgrade: a boosting algorithm that doesn't crumble when real-world data introduces outliers or high-leverage points. The authors have made the paper available open-access on arXiv with code links, so teams can test MM-XGBoost against their own noisy datasets to see if it beats their current gradient-boosting baseline.
- Standard XGBoost's quadratic loss is vulnerable to vertical outliers and leverage points, and Huber loss doesn't fully solve it
- MM-XGBoost uses a two-step MM-estimator, combining M-, S-, and τ-estimators to balance robustness and accuracy
- The paper includes 30 pages plus 15 supplementary, with experiments showing MM-XGBoost offers the best robustness-accuracy trade-off
Why It Matters
A robust XGBoost variant reduces model failure on messy real-world data with outliers, improving trust in gradient boosting.