DomainPilot boosts LLM fine-tuning by 3.8% with no extra data cost
New two-stage framework uses domain-level loss to optimize data mixture without extra compute
Optimizing training data mixtures is a major bottleneck in large language model fine-tuning. Traditional methods like data selection are too expensive for terabyte-scale corpora, while mixture optimization often introduces I/O bottlenecks or requires training auxiliary reference models. Sample-level reweighting is noisy and conflates difficulty with novelty.
DomainPilot solves this with a lightweight patch-based architecture that injects domain-aware loss computation into existing frameworks (e.g., MindSpeed/Megatron-LM) using just ~30 lines of code. It first uses a Scaling Law guided coarse optimization to fit domain-specific convergence curves, then a Mixing Law guided fine optimization to model cross-domain interactions. On Qwen3-1.7B, the optimized mixture delivered consistent gains across reasoning, coding, and function calling benchmarks with zero additional cost.
- DomainPilot achieves +3.8% on LiveCodeBench v5 and +3.6% on BFCL v3 without increasing total data volume or training cost
- Two-stage optimization combines Scaling Law guided coarse tuning with Mixing Law guided fine tuning using domain-level loss signals
- Integration requires only ~30 lines of adapter code for frameworks like MindSpeed/Megatron-LM, avoiding expensive data selection or auxiliary models
Why It Matters
Smarter data mixing cuts costs while improving LLM performance, making fine-tuning more accessible.