Transformers learn abstract rules first, then local patterns – new study

A new study from Wang, Jenkins & Wonnacott applies a developmental approach to understand how neural language models (NLMs) learn statistical patterns. They trained a series of Generative Transformer models on a synthetic grammar and saved model states at multiple checkpoints during training. By analyzing how internal representations evolved, they discovered a clear learning trajectory: NLMs first acquire the most abstract, global statistical regularities, and only later refine their knowledge to capture local dependencies.

This finding mirrors theories of human language development, where infants initially detect broad patterns before honing in on finer details. The models exhibited over-generalizations early in training, which gradually became constrained as learning progressed. The authors propose a new framework to explain NLM statistical learning and language cognition based on this developmental path. The paper received an oral presentation at the Interdisciplinary Advances in Statistical Learning conference and provides concrete evidence of how Transformers build up their linguistic knowledge from abstract to specific.