Two-Stage Optimizer-Aware Online Data Selection for Large Language Models
A new 'Filter-then-Weight' method selects the most useful training data on the fly, cutting costs.
A team of researchers, including Fangxin Wang, Peyman Baghershahi, and Philip S. Yu, has published a paper introducing a novel framework for making large language model (LLM) fine-tuning significantly more data-efficient. The core problem they address is that existing gradient-based data selection methods are designed for offline settings, where all data is available upfront. In real-world online fine-tuning, data arrives sequentially, and the utility of a sample depends on the model's current state and the optimizer's adaptive parameters (like Adam or SGD with momentum). Their key insight is to treat online selection not as a static ranking task, but as a dynamic problem of shaping the model's next update to best match a desired target.
To make this practical, the team developed a two-stage 'Filter-then-Weight' algorithm. The first stage filters a batch of incoming data to identify candidates that are geometrically aligned with the desired update direction. The second stage then optimizes the weighting coefficients for the selected samples, accounting for interactions and redundancy between them. A major technical innovation is a factorized outer-product gradient representation, which enables efficient computation even with long-context data typical for modern LLMs. Experiments demonstrate that this optimizer-aware framework consistently outperforms existing online selection baselines, leading to faster convergence and better downstream task performance without increasing the amount of data used.
- Proposes an 'optimizer-aware' framework that dynamically selects data based on the model's current training state and optimizer geometry.
- Introduces a practical 'Filter-then-Weight' algorithm with efficient computations for long-context LLMs, using factorized gradient representations.
- Shows consistent improvements in convergence speed and final model performance over other methods when using the same data budget.
Why It Matters
This could drastically reduce the cost and time required to fine-tune and continuously improve large models like GPT-4 or Llama 3 on new data streams.