Old-school metrics outsmart deep learning on Spanish speech intelligibility

A new paper from Iván López-Espejo (University of Granada) and Jesper Jensen (Aalborg University) tackles a critical gap in speech technology: how well objective intelligibility metrics (OIMs) generalize to non-English languages. The authors evaluated seven popular metrics on SpInt, a newly curated Spanish speech dataset. The reference-based metrics (STOI, ESTOI, STGI, HASPI, SIIB) require a clean reference signal to compute scores, while the two deep learning-based no-reference metrics (MOSA-Net+ and W2V-SIP) try to predict intelligibility without a reference. The study reveals a striking performance gap: traditional reference-based methods consistently outpaced modern data-driven approaches across all conditions.

The most dramatic failures occurred under acoustic mismatch—specifically when models trained on English data were tested on Spanish speech. MOSA-Net+ and W2V-SIP degraded significantly, showing that current no-reference metrics lack language robustness. This is particularly concerning given the global push toward multilingual voice assistants and hearing aid apps. To address this, the authors are releasing SpInt publicly, hoping to catalyze development of more generalizable no-reference OIMs. The findings suggest that while deep learning excels in many speech tasks, intelligibility assessment still benefits from tried-and-true signal processing approaches—at least until we train models on truly diverse data.