Speech-LLM uses chain-of-thought reasoning to slash rare-word errors in ASR

A team of researchers from KU Leuven has introduced a new method for improving automatic speech recognition (ASR) on rare, domain-specific terms. Their approach, detailed in a paper accepted at Interspeech 2026, uses broad contextual descriptions—such as video metadata—as weak semantic priors. Instead of relying on keyword lists or phrase biases, they train a speech-LLM to perform metadata-driven reasoning chains. The key innovation is a chain-of-thought process: the model first generates an initial transcript, then reasons over the available context, and finally produces a corrected transcript. This forces the model to use external knowledge to fix errors on names and rare terms.

To enable this, the team built a 400-hour dataset of reasoning-augmented speech data derived from YouTube videos. They paired erroneous ASR hypotheses with video metadata and LLM-generated explanations that justify the necessary corrections. Speech-LLM was then finetuned on this data. On held-out YouTube test sets, the method significantly reduced word error rates, especially for rare words and named entities. The work lays the groundwork for deeper contextual reasoning in ASR systems, moving beyond shallow biasing toward true understanding of context.