Crowdsourced audio testing rivals lab quality with smarter screening

Subjective listening tests remain the gold standard for evaluating speech and audio codecs, but traditional lab-based tests are expensive and slow. Crowdsourcing offers a faster, cheaper alternative, but its results have historically suffered from lower reliability due to uncontrolled environments and inattentive participants. A new study by Treffehn et al., accepted at Interspeech 2026, systematically compares the ITU-T P.808 crowdsourced test with the conventional P.800 Degradation Category Rating (DCR) test, using both classical (e.g., Opus) and neural codecs (e.g., Lyra, SoundStream).

The researchers conduct a statistical analysis of various screening methods designed to filter out low-quality responses. They find that post-screening techniques—like verifying that listeners ordered anchors correctly and checking rating span consistency—significantly boost the correlation between crowdsourced and lab results. Continuous screening methods, such as inserting trap questions (e.g., repeated identical stimuli) and gold standard questions (known reference samples), further enhance data quality by detecting inattentive listeners in real time. The study proposes a combined set of these screenings that delivers lab-like fidelity at a fraction of the cost, enabling scalable, bias-free subjective testing for codec development.