Gumbel-BEARD adapts Whisper with 92% less labeled data

Speech foundation models like OpenAI's Whisper struggle in low-resource domains due to domain mismatch and data scarcity. A new paper accepted at Interspeech 2026 introduces Gumbel-BEARD, an end-to-end framework that automates which encoder layers to use during domain adaptation. The key innovation is a hard Gumbel-Softmax selector that learns to pick the most relevant layers for the target domain, paired with a BEST-RQ self-supervised objective that dynamically adapts to local acoustic characteristics. This eliminates the need for manual layer tuning, a common bottleneck in deploying large models on niche data.

The results are striking. On the MyST child speech corpus, using only 10 hours of labeled data for fine-tuning yields word error rates (WERs) that match a fully supervised baseline trained on the entire 133-hour labeled set. Gumbel-BEARD sets new state-of-the-art WERs of 8.21% with Whisper-medium on MyST and 11.06% with Whisper-small on the OGI Spontaneous dataset. On CORAAL, which features adult dialectal speech, the method achieves up to 6% relative WER reduction compared to standard fine-tuning. This demonstrates robust generalization across age groups and dialects, making Gumbel-BEARD a promising approach for any low-resource speech application, from clinical voice analysis to field linguistics.