LassoFlexNet: Flexible Neural Architecture for Tabular Data
A new neural architecture finally challenges XGBoost's dominance on tabular data with built-in interpretability.
A team of researchers has introduced LassoFlexNet, a new neural network architecture specifically engineered to solve deep learning's long-standing weakness with tabular data. While neural networks excel at vision and language tasks, they have consistently underperformed compared to tree-based models like XGBoost and Random Forests on structured datasets. LassoFlexNet bridges this gap by embedding five critical inductive biases—including robustness to irrelevant features and handling feature heterogeneity—directly into its design. Its core innovation is using Per-Feature Embeddings to evaluate each input's contribution and a Tied Group Lasso mechanism to sparsely select only the most relevant variables, enhancing both performance and interpretability.
Training such a complex, sparsity-inducing model presented a significant optimization challenge, as standard methods proved unstable. To solve this, the team developed a novel optimizer called Sequential Hierarchical Proximal Adaptive Gradient with exponential moving averages (EMA) to ensure stable convergence. The results are compelling: across a rigorous test of 52 datasets from three major benchmarks, LassoFlexNet matched or surpassed the performance of state-of-the-art tree-based models. It achieved up to a 10% relative gain in performance, a significant margin in machine learning benchmarks. Crucially, it does this while maintaining the kind of clear, Lasso-like interpretability that is often a key advantage of simpler linear models, offering a 'best of both worlds' solution for data scientists.
- LassoFlexNet incorporates five key inductive biases for tabular data, addressing deep learning's traditional weaknesses like sensitivity to irrelevant features.
- The architecture uses a Tied Group Lasso mechanism for sparse feature selection, providing built-in interpretability similar to linear models.
- On 52 benchmark datasets, it matches or outperforms top tree-based models, achieving up to a 10% relative performance gain.
Why It Matters
This provides data scientists a powerful, interpretable neural alternative to XGBoost for critical tabular data tasks in finance, healthcare, and analytics.