New LLM Sparsity Prior Makes Feature Selection Robust to Bad Weights
LLMs can prioritize features, but bad weights hurt. LSP fixes that.
Get AI news that actually matters
One email a day. Zero fluff. Join 10,000+ professionals.
Large language models offer a scalable way to inject domain knowledge into high-dimensional feature selection, but existing methods like LLM-Lasso suffer when LLM-generated weights are inaccurate. To solve this, the authors first develop a formal framework to quantify the quality of those weights, allowing rigorous evaluation across varying weight regimes.
Then they propose the LLM Sparsity Prior (LSP), which incorporates LLM weights into Spike-and-Slab and Spike-and-Slab Lasso models using two interpretable hyperparameters controlling global sparsity and weight concentration. Hierarchical hyperpriors let the model dynamically discount misleading weights while preserving gains from accurate ones. On a private medical dataset for Acute Kidney Injury, LSP boosted prediction accuracy and uncovered clinically relevant features missed by baselines, showing particular strength in low-data settings and robustness to prompt variation.
- LSP uses two interpretable hyperparameters for global sparsity and weight concentration to dynamically discount misleading LLM weights.
- Includes a novel framework for quantifying the quality of LLM-generated weights, enabling rigorous evaluation across regimes.
- On a private Acute Kidney Injury dataset, LSP outperformed baselines in prediction accuracy and feature discovery, especially with limited training data.
Why It Matters
Enables reliable AI-driven feature selection in critical fields like medicine, even when LLM priors are noisy.