VisG AV-HuBERT: Viseme-Guided AV-HuBERT

A research team from Trinity College Dublin and Stanford University has introduced VisG AV-HuBERT, a novel framework designed to make AI lip-reading and speech recognition systems far more robust in noisy environments. The core innovation is a multi-task fine-tuning approach that adds an auxiliary task of viseme classification to the established AV-HuBERT model. Visemes are the visual equivalent of phonemes—the basic units of visible speech, like the lip shape for "p" or "b." By forcing the model's encoder to simultaneously predict these visual speech units, it learns to preserve and rely more heavily on the visual articulatory features from video, strengthening the audio-visual fusion at a foundational level.

Evaluated on the challenging LRS3 dataset, the results are striking, particularly under acoustic duress. In heavy Speech noise at a -10 dB Signal-to-Noise Ratio, VisG AV-HuBERT reduced the Word Error Rate from 13.59% to just 6.60%—a 51.4% relative improvement over the baseline. The model also demonstrated strong generalization on the LRS2 dataset. A deeper error analysis revealed substantial reductions in substitution errors (e.g., mistaking "bat" for "pat") across different noise types, indicating the model has genuinely improved its discrimination of subtle speech units by leveraging the visual stream more effectively.

This work, accepted for publication at ICPR 2026, provides a clear pathway for enhancing noise-robust Audio-Visual Speech Recognition (AVSR). It demonstrates that explicit, encoder-level guidance using visual linguistic features (visemes) can yield significant gains, separate from just improving the language model decoder. The approach is relatively lightweight, adding only a small sub-network, making it a practical upgrade for real-world applications where audio quality is unreliable, such as video conferencing, assistive hearing technology, and in-car voice assistants.