Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor\`ub\'a

A new research paper from the University of Edinburgh, accepted at Speech Prosody 2026, identifies a critical flaw in how current AI models process speech. The study, led by Opeyemi Osakuade and Simon King, probes Discrete Speech Units (DSUs)—compressed representations derived from models like Wav2Vec 2.0 or HuBERT that are used for building efficient text-to-speech and multimodal dialogue systems. Their investigation into the tonal languages Mandarin and Yorùbá reveals that while the initial self-supervised learning (SSL) latent representations do encode lexical tone, the subsequent quantization step (like K-means clustering) strips this information out, prioritizing segmental phonetic structure instead.

This failure to capture 'suprasegmental' features like tone is a major roadblock for creating accurate, natural-sounding AI for the billions of speakers of tonal languages. An AI voice system using current DSUs could pronounce Mandarin words with correct phonemes but the wrong tone, completely changing their meaning. The researchers tested multiple quantization methods and found the problem persists, indicating it's a fundamental limitation of the current approach.

The authors don't just diagnose the problem; they also point toward a solution. They demonstrate that a two-stage quantization process—first clustering for phonetic information, then clustering the residual representation—can better preserve tonal data. This work signals a necessary shift in speech AI research, highlighting that future models must be explicitly designed to be 'tone-aware' or 'prosody-aware' to achieve true global utility and avoid miscommunication in critical applications.