New AI framework M2S-AVSR boosts speech recognition by 29.4% in noisy environments

Audio-visual speech recognition (AVSR) combines audio and visual cues to improve accuracy, but real-world conditions like camera angle changes, background noise, or partial face occlusion often break the system. A new paper from researchers introduces M2S-AVSR, a framework designed to handle exactly these challenges. The model has two key innovations: a multi-view representation learning encoder that learns visual speech features invariant to viewpoint changes, and a modality-aware module that explicitly models the quality of each input and cross-modal synchrony. This allows the system to dynamically weight the audio and visual streams and inject fine-grained visual information during the decoding stage, making it robust even when one modality is degraded.

Tests on English (LRS3) and Mandarin (MISP2021-AVSR) benchmarks show dramatic gains. Under viewpoint perturbation and visual degradation, M2S-AVSR delivers up to a 29.4% relative improvement over previous methods. It also achieves new state-of-the-art results on the MISP2021-AVSR test set. The paper also introduces AISHELL8-RealScene, a public multi-scenario, multi-view conversational AV dataset recorded in real-world environments, establishing a new benchmark for outdoor and challenging scenes. On this dataset, M2S-AVSR achieves the best results, particularly in outdoor scenarios where lighting, motion, and audio noise are unpredictable. The work has been submitted to IEEE Transactions on Audio, Speech, and Language Processing.