New study: Modern ASR systems break perplexity-error rate link

For decades, language model perplexity (PPL) has served as a convenient proxy for automatic speech recognition (ASR) word error rate (WER), with prior work reporting an approximately linear relationship when both metrics are plotted in log-log space. However, modern end-to-end ASR systems—architectures like attention-based encoder-decoders that already embed language modeling capacity internally—challenge this assumption. In a new paper submitted to SLT 2026, Mohammad Zeineldeen and co-authors systematically test whether the PPL-WER relation still holds for these contemporary models. They evaluate systems with external neural LMs and large language models (LLMs), varying encoder context lengths, and performing internal language model (ILM) subtraction. The paper provides experimental evidence that the traditional linear trend breaks down, especially when ILM subtraction is applied, and that LLM perplexities do not align with standard neural LM trends. This forces ASR researchers to reconsider using PPL as a trustworthy indicator of end-to-end system quality.

The study also investigates whether external language models still deliver improvements in modern end-to-end setups. While external LMs remain beneficial, the extent of their impact depends heavily on how the system’s internal LM is accounted for. ILM subtraction—a technique that removes the decoder’s internal language modeling to isolate external LM contributions—substantially changes the observed PPL-WER relation, implying that the decoder’s own language modeling must be factored in when interpreting external LM quality. Additionally, the team shows that encoder context length influences the relationship: longer contexts reduce the effect of external LMs, as the encoder itself captures more linguistic structure. For practitioners integrating LLMs into speech pipelines, the findings imply that perplexity alone cannot predict final WER gains. The paper offers a new baseline for evaluating ASR systems that blend internal and external language models, and calls for more nuanced evaluation metrics.