DataPrep-Bench: New benchmark tests LLMs as data preparers

A team of researchers led by Hao Liang has released DataPrep-Bench, the first comprehensive benchmark to evaluate how well large language models (LLMs), agents, and data-centric workflows prepare training data end to end. The benchmark splits the task into two complementary capabilities: data construction, which transforms raw sources into supervised training sets, and data quality evaluation, which predicts a candidate dataset's downstream training value without requiring full fine-tuning. DataPrep-Bench uses a shared downstream-grounded protocol across six diverse domains (including Math, Science, and Medical) and multiple base models, with Llama-3.1-8B as a key test model.

For data construction, the team also introduces Data-Construction-Skill, a skill-guided agent that lifts the Dolly-15k baseline by nearly 20 absolute points on Llama-3.1-8B Finance and matches top agent- and DataFlow-based methods in knowledge-extraction-dense domains. For quality evaluation, they propose the Distributional Alignment Score (DAS), which uses Maximum Mean Discrepancy (MMD) between a candidate dataset and a domain proxy. DAS achieves the strongest cross-model correlation in four of six domains and is the only metric to clear r > 0.70 simultaneously in Math, Science, and Medical, outperforming existing quality-, diversity-, and heuristic-based evaluators. The benchmark provides a unified framework for measuring progress on both capabilities as co-equal targets of LLM-driven data preparation.