New LLM approach cuts multi-talker speech errors by 24%

Researchers from an undisclosed institution (authors Naijun Zheng, Yuke Lin, Sanli Tian, et al.) have introduced a novel method for multi-talker speech recognition that leverages large language models (LLMs) without requiring massive annotated multi-talker datasets. Their system, accepted at Interspeech 2026, uses a dual-encoder architecture—one encoder extracts semantic features for ASR, the other captures speaker characteristics for diarization. These features are interleaved in a custom format before being fed into the LLM, enabling it to jointly model who said what. Two key training innovations are introduced: a length-aware speaker ID loss that improves diarization accuracy by penalizing mismatches proportional to utterance duration, and an adaptive threshold strategy for ASR loss computation that dynamically downweights regions of speech overlap, mitigating the model's tendency to hallucinate extra words.

The combined approach yields significant gains on real-world meeting and multi-talker corpora. On the AliMeeting corpus, the system outperforms open-source baselines by 18% relative in word error rate combined with diarization error rate, and on Aishell4 by 24% relative. Importantly, these improvements come without requiring synthetic data or costly human annotation of overlapping speech—the authors train only on available real-recorded data with standard speaker labels. This work demonstrates that careful architectural design and loss balancing can make LLM-based multi-talker ASR practical even with limited data, addressing a key bottleneck in deploying such systems for meetings, call centers, and ambient intelligence applications.