Tracking the emergence of linguistic structure in self-supervised models learning from speech

A team of computational linguists from the University of Amsterdam has published a foundational study tracking exactly how and when AI speech models learn the building blocks of language. By analyzing six different Wav2Vec2 and HuBERT models trained on thousands of hours of spoken Dutch, the researchers mapped the emergence of linguistic structures—from basic phonemes and syllables to complex syntax and semantics—across 12 neural network layers and throughout the training process. They discovered that different linguistic features follow distinct learning trajectories, largely determined by their abstraction from raw audio and the timescale of information integration.

Crucially, the study reveals that the design of the model's pre-training objective dramatically shapes how it organizes language. Models like HuBERT, which use higher-order prediction tasks involving iteratively refined pseudo-labels, showed greater parallelism in learning multiple linguistic structures simultaneously across layers. In contrast, models with simpler masking objectives exhibited more sequential, layer-by-layer specialization. This research provides the first comprehensive map of linguistic emergence in self-supervised speech models, offering concrete engineering insights for building AI that processes spoken language more efficiently and transparently.