Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios with Synthetic Visual Data

A team from Universitat Politècnica de Catalunya has developed a novel framework that enables Audiovisual Speech Recognition (AVSR) for languages with absolutely no annotated video training data. Their method addresses a critical bottleneck in AVSR development: while combining audio and visual cues (like lip movements) significantly improves transcription accuracy, especially in noisy environments, creating labeled video corpora for training is prohibitively expensive for most of the world's languages. The researchers' solution is to bypass the need for real video entirely.

They created a scalable synthetic data pipeline that generates realistic talking-head videos. The process starts with static facial images and real audio recordings in the target language. Using lip-syncing technology, they animate the images to match the audio, producing over 700 hours of synthetic video for Catalan—a language with no existing annotated AV corpus. This synthetic data was then used to fine-tune AV-HuBERT, a powerful pre-trained multimodal model.

The results are striking. On a manually annotated Catalan benchmark, their model achieved performance close to state-of-the-art systems trained on massive real video datasets, despite using far fewer parameters and training resources. Crucially, it outperformed an identically trained audio-only model and preserved the key advantage of multimodal systems: robustness against acoustic noise. This demonstrates that synthetic visual data can be a viable and effective substitute for real recordings.

This research, submitted to Interspeech 2026, represents a major step toward democratizing advanced speech technology. By removing the dependency on scarce, expensive video data, the framework opens the door for developing robust AVSR systems for hundreds of under-resourced languages and dialects. It showcases how generative AI can be used not just as an end product, but as a tool to bootstrap the creation of other, more specialized AI systems where data is the primary constraint.