SLAP reduces LLM training data by 40% with smarter selection
The assumption that more data always yields better models is being overturned: a new framework called SLAP reduces instruction tuning datasets by up to 40% while maintaining performance, challenging foundational beliefs about data hunger in AI.
SLAP (Stratified Loss-based Pruning) introduces a novel batch-aware data selection method for efficient instruction tuning of large language models. Traditional approaches evaluate individual data points, but SLAP instead assesses the learnability of entire batch compositions. It uses distribution-aware stratified sampling to ensure coverage of the full data distribution, then maximizes intra-batch diversity through relative distance optimization. By leveraging Hessian-approximated gradient information, SLAP dynamically selects the most informative batches during training.
Tested across multiple architectures (LLaMA, ChatGLM) and downstream tasks (multi-turn dialogue, multilingual translation, question answering), SLAP consistently outperforms existing state-of-the-art methods. Most notably, it achieves superior results using only 60-80% of the training data compared to full dataset training, reducing computational costs substantially. This breakthrough makes high-quality instruction tuning more accessible, especially for teams with limited compute budgets.
- SLAP reduces instruction tuning data by 20-40% without performance loss, saving compute and storage costs.
- The method uses batch-aware stratification based on loss gradients and Hessian approximations, outdoing static dataset curation approaches.
- Adoption depends on managing computational overhead of Hessian-vector products and verifying generalizability across architectures.
Why It Matters
Efficient data selection could democratize LLM fine-tuning, lowering costs and accelerating AI iteration cycles.