Gradient boosting gets vector-valued leaves for multi-class efficiency
David Cortes' new algorithm avoids diagonal approximations in multi-class boosting.
Deep Dive
Gradient boosting with vector-valued leafs: David Cortes' new paper extends gradient boosting to handle vector-valued objective functions, like multinomial log-likelihood for multi-class classification. Instead of updating one class at a time or using diagonal upper bounds on the Hessian, the proposed algorithm works on full vector inputs using histogram-based decision trees.
Key Points
- Extends gradient boosting to vector-valued leaf outputs, avoiding per-class updates.
- Replaces diagonal Hessian approximations with full second-derivative matrices for better accuracy.
- Compatible with histogram-based decision trees, preserving scalability for large datasets.
Why It Matters
Boosts multi-class and multi-output gradient boosting accuracy without sacrificing speed.