Data-DPO picks smarter SFT data, beating full dataset training
Probing activation gaps, Data-DPO selects data that beats full-scale fine-tuning.
Data selection in supervised fine-tuning (SFT) typically treats data value as a static property, ignoring how well samples fit the target model's current capabilities. To fix this, researchers including Peng Sun, Kai Yu, and Lu Chen introduce Data-DPO, a method that explicitly tailors data selection to the target model. Data-DPO runs a one-step probe on candidate samples, observing the local training feedback as activation differences between samples. These differences are then transformed into pairwise data preferences, which train a lightweight reward model to learn what data is most useful for that specific model.
This target-model-aware reward is then combined with external quality scores and a marginal diversity term to construct the final training subset. Experiments on Vision-Flan and LLaVA-CoT benchmarks show that Data-DPO consistently outperforms existing data selection baselines across multiple data budgets, and stably surpasses full-data training performance. The paper is available on arXiv (2608.16926), offering a practical, computationally efficient path to reduce training costs while maintaining—or even improving—model quality.
- Data-DPO uses one-step probing to convert activation differences into pairwise data preferences for the target model.
- A lightweight reward model learns target-model-aware preferences; combined with quality scores and marginal diversity, it builds stable training subsets.
- Outperforms existing baselines on Vision-Flan and LLaVA-CoT, stably surpassing full-data training across multiple data budgets.
Why It Matters
Data-DPO cuts SFT costs by selecting only the most effective samples, potentially accelerating LLM fine-tuning pipelines.