Researchers slash speaker ID errors by 41% for coughs and laughs

Current speaker verification (SV) systems excel on regular speech but fail on non-verbal vocalizations (NVVs) like laughter, coughs, sighs, and grunts—sounds increasingly generated by expressive TTS and voice conversion. Fine-tuning SV models on NVV data causes catastrophic forgetting of speech performance. To solve this, a team from National Taiwan University, Academia Sinica, and USC proposed a conditional distillation and mixture-of-experts approach, accepted at INTERSPEECH 2026.

Their framework freezes a Data2Vec self-supervised feature extractor, feeds it into an ECAPA-TDNN backbone, and adds a Mixture of Experts (MoE) module with learned domain-aware routing to handle diverse NVV types. A conditional distillation loss from a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss reduces the domain gap between speech and NVVs. Across 10 NVV types, the method reduced speech-NVV equal error rate (EER) from 38.93% to 22.66%, and improved speech EER from 13.17% to 9.24%, demonstrating a practical path to unified speaker verification for expressive AI voices.