How a Simple Change in Training Data Made Speech AI Understand 60 Languages Better — And Why Location Matters

Self-supervised speech encoders like Whisper and Wav2Vec2 are typically fine-tuned with language-level supervision, which can overlook crucial geographical variation within a language. A new preprint (arXiv:2606.19940) from Pavan Kumar J and colleagues tackles this by fine-tuning both models on 60 Indic languages with two supervision regimes: language-only (60 classes) and joint language-district (386 classes, combining language with Indian districts). Using Normalized Conditional Mutual Information (NCMI), they analyzed the structure of learned embeddings.

Results show that joint language-district supervision creates global language clusters with well-organized within-language subclusters aligned to district variation. This significantly improves district discrimination conditioned on language—i.e., the model can better distinguish between speakers from different regions of the same language—while maintaining strong marginal language classification performance. For practical speech applications in India, this means more accurate accent and dialect handling without sacrificing multi-language support.