Data Mixing for Large Language Models Pretraining: A Survey and Outlook

A team of researchers has published a seminal survey paper titled 'Data Mixing for Large Language Models Pretraining: A Survey and Outlook,' providing the first dedicated, systematic review of this critical but fragmented area of AI research. The paper formalizes data mixture optimization as a bilevel problem, clarifying its role in the pretraining pipeline for models like GPT-4 and Llama 3. Unlike sample-level selection, data mixing focuses on optimizing domain-level sampling weights—such as how much code versus web text to use—to allocate limited compute and data budgets more effectively for better model generalization.

The authors introduce a fine-grained taxonomy that organizes existing methods along two primary dimensions: static mixing (rule-based or learning-based) and dynamic mixing (adaptive or externally guided). For each class, they analyze the performance-cost trade-offs of representative approaches. The survey critically highlights major cross-cutting challenges, including the limited transferability of optimal mixtures across different data domains, model architectures, and validation sets, as well as unstandardized evaluation protocols.

Finally, the paper outlines promising future research directions, such as finer-grained domain partitioning, inverse data mixing, and pipeline-aware designs. This work consolidates scattered knowledge into a single framework, aiming to provide conceptual and methodological insights that could lead to more efficient training of the next generation of LLMs, potentially reducing costs and improving capabilities across the board.