Researchers expose bias in lexicon evaluation metrics for zero-resource speech

Building a lexicon from discovered word-like units is a central goal of zero-resource speech processing—but the metrics we use to evaluate those lexicons may be deeply flawed. In a new paper from Stellenbosch University, Simon Malan, Danel Slabbert, and Herman Kamper systematically analyze normalized edit distance (NED), the most common evaluation metric. They demonstrate that NED has an inherent bias toward large clusters: because it averages phoneme edit distances across all clusters, poorly formed small clusters are drowned out by well-formed large ones. The metric also ignores how well true word classes are distributed across clusters, meaning a lexicon could appear high-quality while actually lumping distinct words together.

To address these shortcomings, the authors propose two new metrics grounded in established clustering theory. The first is a modified within-cluster consistency metric that weighs each cluster’s contribution by its size, preventing large clusters from dominating the score. The second is an inverse metric that measures how true word classes are spread across clusters—penalizing cases where a single class is scattered across many discovered clusters. Through experiments on both synthetic lexicons and real-world speech data, the authors show that these combined metrics: (1) correlate significantly better with how similar a lexicon is to the ground-truth distribution, and (2) are far more robust to the biases that skew NED-based evaluations. The work includes 6 figures and is available on arXiv (2606.06183).