New multimodal framework boosts ASR and dialect ID for 33 Indian dialects

Researchers Saurabh Kumar, Amartyaveer, and Prasanta Kumar Ghosh from the Indian Institute of Science (IISc) have published a paper on arXiv detailing a multimodal framework that jointly optimizes Automatic Speech Recognition (ASR) and Dialect Identification (DID) for Indian languages. The approach addresses a critical gap: existing methods optimize ASR or DID individually, leading to performance trade-offs in low-resource, dialect-rich settings. The proposed system uses a Bottleneck Encoder to extract dialectal features from Conformer-based speech representations, while a RoBERTa encoder processes ASR-generated CTC embeddings. A gating mechanism merges these features, followed by an attention encoder that refines the representations. The learned embeddings are then concatenated with Conformer outputs to enhance ASR features.

Evaluated on eight Indian languages covering 33 distinct dialects, the method achieves an average DID accuracy of 81.63%, a Character Error Rate (CER) of 4.65%, and a Word Error Rate (WER) of 17.73%. These results demonstrate significant improvement over models that handle ASR and DID separately, especially for low-resource dialects. The work is particularly relevant for India's linguistic diversity, where dialectal variation often degrades speech recognition quality. By jointly learning to identify dialects and transcribe speech, the framework could enable more inclusive voice interfaces and accessibility tools across the subcontinent.