GEM: New Data Curation Method Boosts LLM Accuracy by 1.2%

A new paper from Yue Min and colleagues presents GEM (Geometric Entropy Mixing), a framework that treats LLM pre-training data curation as a variational optimization problem on the hypersphere. Traditional methods suffer from ontological misalignment in human taxonomies and fail to address embedding anisotropy in Euclidean clustering. GEM counters this by decoupling the generative prior and using a Minorize-Maximize (MM) algorithm to optimize a mixing-balance regularizer, effectively counteracting cluster collapse. To scale to web-scale corpora, the authors employ teacher-student distillation, and they introduce the Geometric Influence Score (GIS) for generating interpretable taxonomies. This geometric approach discovers balanced semantic structures that Euclidean heuristics miss.

Experiments with 1.1B-parameter models demonstrate that GEM establishes a new state-of-the-art when integrated into existing mixing strategies like DoReMi and RegMix. The method improves average downstream accuracy by up to 1.2%, offering a robust coordinate system for predictable data mixing. This work has significant implications for LLM training efficiency: better data composition means models can achieve higher performance without increasing data volume. The paper is submitted to ICML 2026 and available on arXiv.