Parallel Gradient Boosting speeds up multi-output predictions by orders of magnitude

Gradient boosting, one of the most successful learning techniques, struggles with multi-output problems like predicting entire conditional distributions (e.g., multiple quantile regression). Classical implementations train one base model per target at every iteration, making them computationally expensive. Researchers propose Parallel Gradient Boosting, which uses a common descent direction for all training observations. This allows only one base model per iteration regardless of the number of targets, yielding dramatic speedups. The authors prove sufficient conditions for convergence and demonstrate the algorithm on multiple quantile regression. Results show prediction quality equal to XGBoost but with orders-of-magnitude faster training.

Beyond speed, the estimator excels in challenging real-world scenarios: high-dimensional feature spaces and datasets with mixed or missing covariates. It outperforms both nonparametric (e.g., kernel-based) and semiparametric (e.g., quantile regression forests) methods. This technique makes large-scale probabilistic forecasting practical, with applications in finance (risk estimation), healthcare (patient outcome distributions), and climate modeling. The shared descent direction elegantly decouples the training cost from the number of target quantiles, enabling efficient, flexible estimation without sacrificing performance.