New study warns ASR metrics may overestimate speech enhancement quality

A new study from the University of Hamburg, led by Danilo de Oliveira, Tal Peer, and Timo Gerkmann, and published on arXiv (2605.12107), challenges the common practice of using automatic speech recognition (ASR) word error rate (WER) as a proxy for evaluating speech enhancement (SE) systems. The researchers conducted a listening experiment to compare how modern ASR models correlate with human recognition of enhanced speech. They tested several ASR architectures, including transformer-based models and a transducer model, and found that those with large-scale noisy training and embedded language models (e.g., transducer-based systems) produce WER scores that align more closely with human performance. However, this correlation comes with a critical caveat: the same robustness that makes these ASR models reliable also makes them less sensitive to the purely acoustic improvements that SE is meant to deliver. Because these models can “fill in” missing phonemes using language context and are trained on noisy speech, they may report low WER even when the enhanced signal is still acoustically degraded. This means that a WER improvement after SE could be misleading – the SE system might not actually be cleaning up the audio; the ASR model is just better at guessing the correct words from context. The authors argue that relying solely on WER from modern ASR can give an inflated impression of SE quality, and recommend complementing such metrics with acoustic-focused measures (e.g., PESQ, STOI) and listening tests.

This has immediate implications for the speech processing community: researchers and engineers who use ASR-based evaluation to benchmark their SE algorithms might be overestimating their systems' real-world performance. For applications like hearing aids, voice assistants, or teleconferencing, where clean audio is critical, trusting WER alone could lead to deploying SE solutions that don't actually enhance speech intelligibility for humans. The study suggests a hybrid evaluation pipeline using both a high-performance transducer ASR for WER and acoustic metrics that capture signal-level quality. The paper also highlights that simpler, non-contextual ASR models (e.g., CTC-based with no language model) can sometimes be more honest about poor acoustic quality, trading human correlation for transparency. As SE research moves toward deployment, this work underscores the danger of relying on a single metric – even one as intuitive as WER – without understanding the underlying model's behavior.