AVSR models collapse on new test set, exposing overfitting
Five state-of-the-art models fail under strictly matched conditions, and audio-visual even lags audio-only.
A new study led by Zhaofeng Lin (Imperial College London and Trinity College Dublin) systematically assesses the true generalisability of Audio-Visual Speech Recognition (AVSR) models. Despite near-perfect scores on the standard LRS3 benchmark, the researchers constructed a highly controlled test set from the massive MultiVSR dataset that strictly matches the acoustic, visual, and demographic distributions of LRS3. When evaluating five state-of-the-art AVSR architectures on this matched set, every model suffered a universal performance collapse, proving that current systems fail to generalise even under aligned conditions.
The paper presents a fine-grained attribute analysis across seven factors, isolating specific drivers of degradation. Notably, the team uncovered a profound lexical bias and distinct error patterns, and surprisingly revealed that audio-visual performance lags behind audio-only settings. The findings strongly suggest adaptive overfitting to the LRS3 benchmark rather than true learning of multimodal speech patterns. The authors release their matched test set to enable more rigorous benchmarking in future AVSR research. The paper has been accepted to Interspeech 2026 as a long paper.
- Five top AVSR architectures (e.g., from Meta, Google) all collapse on a new matched test set, confirming adaptive overfitting to LRS3.
- Audio-visual performance lags behind audio-only baselines, undermining the supposed benefit of visual cues.
- Fine-grained analysis across seven factors isolates lexical bias and distinct error patterns as key drivers of degradation.
Why It Matters
AVSR models may be memorizing benchmarks, not learning generalisable speech understanding—urgent need for robust evaluation.