TAKE framework shrinks text datasets to 0.1% without losing accuracy

Modern NLP increasingly struggles with the cost of storing and training on massive text corpora. A new paper from researchers at Deakin University (Tri-Nhan Vo, Dang Nguyen, Sunil Gupta) introduces TAKE (Trajectory-Aware Knowledge Estimation), a dataset distillation framework that compresses text datasets to as little as 0.1% of their original size—equivalent to just 20 samples per class—while maintaining downstream task fidelity. The method is grounded in influence functions, which quantify each sample's contribution to the training objective. TAKE goes a step further by convolving influence scores along the entire training trajectory into a single per-sample knowledge score, capturing the most informative examples. These scores are then used as weights within a discrete optimal transport objective to guide prototype selection from a synthetically generated candidate pool.

Evaluated on text classification and natural language inference benchmarks, TAKE achieves strong performance even at extreme compression ratios. The approach is theoretically grounded and has broader implications for coreset construction, continual learning, and data-centric AI. The authors have released their source code publicly, enabling further experimentation. This work directly addresses the growing bottleneck of large-scale text data, offering a principled way to dramatically reduce storage and computational costs without sacrificing model quality. For practitioners, TAKE could enable training competitive models on far fewer examples, reducing both energy consumption and data management overhead.