Optimal Transport aligns audio-visual speech for state-of-the-art LLM recognition

Large language model–based audio-visual speech recognition (LLM-AVSR) has proven robust in noisy environments by combining audio and visual cues. However, existing methods fuse independently pretrained encoders without explicitly reconciling the representational gap between audio, video, and text modalities. This paper introduces an optimal transport (OT)-based semantic alignment framework that explicitly bridges this gap before multimodal fusion. The OT mechanism estimates probabilistic coupling matrices that map modality-specific features to linguistic embeddings, serving as soft pseudo-labels for contrastive learning. This forces the model to extract semantically coherent, cross-modal consistent representations anchored to the LLM's language space, enabling more effective fusion and decoding.

Implemented with a Whisper acoustic encoder, an AV-HuBERT visual encoder, and a LLaMA3.2-3B decoder, the framework was evaluated on the LRS3-TED benchmark. It outperformed strong baselines and achieved state-of-the-art performance under both clean and noisy conditions across a wide range of signal-to-noise ratios (SNRs). The results demonstrate that explicit semantic alignment via OT significantly improves robustness and accuracy in AVSR, opening the door to more reliable speech interfaces in real-world acoustic environments.