UBG-Net boosts audio-visual speech recognition with Bayesian noise filtering
New model handles corrupted audio and video better than existing methods
A team of researchers from China has introduced UBG-Net (Uncertainty-aware Bayesian Gating Network), a new framework designed to make audio-visual speech recognition (AVSR) far more robust in real-world environments. The core innovation is a Modal Uncertainty-aware Bayesian Fusion (MUBF) mechanism that injects signal-level aleatoric uncertainty into a Bayesian network, enabling the model to capture both data noise and model uncertainty during fusion of pre-trained audio and visual features.
For decoding, the system uses Distribution Uncertainty-aware Hierarchical Voting (DUHV), which selects the best transcription from Monte Carlo samples by prioritizing frequency and using inference scores as tiebreakers. On the AVCocktail and LRS2 benchmarks, UBG-Net consistently outperforms current state-of-the-art methods. Ablation studies confirm that both MUBF and DUHV contribute significantly to noise filtering and decoding robustness.
- UBG-Net uses a Modal Uncertainty-aware Bayesian Fusion (MUBF) to combine audio and visual features while modeling both aleatoric and epistemic uncertainty
- A Distribution Uncertainty-aware Hierarchical Voting (DUHV) method selects the best transcription from Monte Carlo samples, improving decoding robustness
- Outperforms SOTA baselines on the AVCocktail and LRS2 datasets, with ablation studies confirming the effectiveness of MUBF and DUHV
Why It Matters
More reliable speech recognition in noisy environments, crucial for real-world applications like hearing aids, voice assistants, and video conferencing.