New MASS tool cuts LLM training costs by selecting smarter data

A team of researchers from Shanghai Jiao Tong University, including Peng Sun, Yi Yang, and others, has developed MASS (Manifold-aware Sparse Selection), a novel approach to optimize LLM post-training data selection. The method addresses a critical bottleneck: as supervised fine-tuning datasets grow, selecting high-value subsets becomes essential to reduce computational costs without sacrificing model performance.

MASS tackles this by reformulating data selection as a hierarchical coverage problem. It first learns low-dimensional principal manifold coordinates using a dense autoencoder to group data semantically at a coarse level. Then, within each group, it applies quality-aware sparse feature coverage via a TopK sparse autoencoder to pinpoint the most informative samples. In experiments on Vision Flan and LLaVA-CoT benchmarks, MASS consistently outperformed strong baselines across multiple training budgets. Notably, in several settings, it matched or even surpassed the performance of full-dataset training while using only a fraction of the data—demonstrating significant efficiency gains for LLM fine-tuning pipelines.