Audio-Image Alignment Boosts Low-Resource ASR Without Transcriptions
New method uses images to adapt speech models, cutting transcription costs dramatically.
Deep Dive
Researchers propose a transcription-free adaptation stage for low-resource ASR. They align audio representations from a pretrained FastConformer encoder with image representations from vision encoders using the Vaani dataset, where images serve as speech prompts. After alignment, supervised fine-tuning consistently outperforms direct fine-tuning. This approach highlights the potential of audio-image representation alignment as a transcription-free adaptation strategy for enhancing ASR in low-resource language settings.
Key Points
- Uses Vaani dataset's naturally paired audio-image data (images as prompts for speech collection).
- Alignment stage between large-scale pretraining and supervised fine-tuning avoids any transcription requirement.
- Multiple vision encoders tested; consistent ASR improvements over direct fine-tuning, with up to 15% relative WER reduction.
Why It Matters
Enables ASR for thousands of low-resource languages without expensive transcription labor, democratizing voice interfaces.