Learning Tree-Based Models with Gradient Descent

Sascha Marton's PhD thesis presents a groundbreaking method for training decision trees (DTs), a classic ML model prized for interpretability but notoriously difficult to optimize. Traditional algorithms like CART use greedy search, making locally optimal split decisions at each node, which often leads to suboptimal overall tree structures and prevents integration with modern, gradient-based ML frameworks. Marton's novel approach tackles this by enabling gradient descent training for hard, axis-aligned DTs. The key is using a straight-through operator on a dense DT representation, allowing the use of backpropagation to jointly optimize all tree parameters simultaneously, rather than sequentially.

This gradient-based optimization directly addresses two major limitations of legacy DT algorithms. First, it escapes the constraint of greedy, local decisions by optimizing the entire tree structure at once. Second, it seamlessly integrates DTs into contemporary ML workflows that rely on gradient descent, such as multimodal learning and reinforcement learning. The thesis reports that this method achieves state-of-the-art results across several domains, including interpretable DTs for tabular data and advanced models for complex tasks without information loss. By bridging the gap between the interpretability of tree-based models and the power of gradient-based optimization, this work significantly expands the applicability and performance of DTs in modern machine learning.