New Unified Framework Uses Bregman Divergences to Supercharge Decision Trees
A single convex function generalizes CART to handle diverse loss geometries...
Decision trees are a staple of interpretable machine learning, but their standard implementations (e.g., CART) use handcrafted impurity criteria like Gini or entropy that must be redesigned for each new statistical model. A new paper by Mathias Bourel introduces a unified framework based on Bregman divergences, a broad family of loss functions that includes the squared Euclidean distance, Kullback-Leibler divergence, Poisson divergence, and Itakura-Saito divergence. By replacing ad-hoc splitting rules with a convex optimization-derived criterion, the method allows decision trees to automatically adapt to any exponential-family distribution or information-geometric structure.
The framework also yields theoretical guarantees: strong convexity of the generating function ensures non-negative impurity gains, while smoothness controls the estimator's stability and consistency. This means practitioners can now train decision trees that are both interpretable and statistically principled for tasks like count regression (Poisson), probabilistic classification (KL), or audio processing (Itakura-Saito). The work bridges convex analysis and statistical learning, offering a drop-in replacement for CART that broadens its applicability without sacrificing performance or theoretical soundness.
- Generalizes CART to any Bregman divergence, including KL, Poisson, and Itakura-Saito, enabling custom loss geometries.
- Derives impurity gains from a single convex function, abandoning ad-hoc criteria for a principled convex optimization approach.
- Provides consistency and stability guarantees based on the convex function's strong convexity and smoothness properties.
Why It Matters
Makes decision trees adaptable to any probabilistic model while preserving interpretability and theoretical rigor.