The Power of Power Law: Asymmetry Enables Compositional Reasoning
Training on rare words makes AI smarter at composing knowledge...
In a new paper on arXiv, researchers Zixuan Wang, Xingyu Dang, Jason D. Lee, and Kaifeng Lyu challenge the conventional wisdom that training data should be reweighted toward a uniform distribution to improve learning of rare skills. They found that across compositional reasoning tasks—such as state tracking and multi-step arithmetic—models trained on power-law distributions (which mirror the natural frequency of words and concepts) consistently outperform those trained on uniform distributions. This counterintuitive result suggests that the inherent skew in natural language data may actually be beneficial for teaching AI to compose knowledge.
The team introduced a minimalist skill-composition task to analyze this effect theoretically. Their analysis reveals that power-law sampling creates a beneficial asymmetry in the model's loss landscape, allowing it to first acquire high-frequency skill compositions with low data complexity. These common skills then serve as stepping stones, making it easier for the model to learn rare, long-tail combinations. This work offers a new perspective on optimal data distribution for training models, with implications for improving reasoning capabilities in large language models without requiring more data.
- Power-law data distribution consistently outperforms uniform distribution on compositional reasoning tasks like state tracking and multi-step arithmetic.
- Theoretical analysis shows power-law sampling induces beneficial asymmetry in the loss landscape, enabling efficient learning of high-frequency skills first.
- Rare long-tail skills are learned more effectively because common skills act as stepping stones, reducing total data requirements.
Why It Matters
This insight could improve AI reasoning by leveraging natural data skew, reducing training data needs.