Robust Audio-Visual Target Speaker Extraction with Emotion-Aware Multiple Enrollment Fusion

A team of researchers has published a new paper, 'Robust Audio-Visual Target Speaker Extraction with Emotion-Aware Multiple Enrollment Fusion,' tackling the classic 'cocktail party problem'—isolating a single speaker's voice in a crowded, noisy room. The model goes beyond traditional audio-only methods by intelligently fusing multiple visual and audio cues. It uses utterance-level data like a speaker's unique voiceprint and a steady face image, combined with frame-level data like lip motion and, innovatively, facial expressions to gauge emotion. This multi-enrollment approach provides a richer set of signals to identify and extract the target speaker's audio from a messy mix.

The key breakthrough is in robustness. The researchers found that while fusing all these modalities works perfectly in ideal lab conditions, performance plummets in the real world where video signals can freeze or drop. Their solution was to deliberately train the model with a high rate of simulated 'modality missing,' where random visual inputs are dropped during training. This technique dramatically enhanced the system's stability, allowing it to maintain strong performance even under severe, unseen signal loss during testing. They demonstrated that a fusion strategy focusing on one key face frame and lip features offers an optimal balance of high accuracy and resilience.

The work, submitted to Interspeech 2026, is fully open-source, with the model and code publicly shared. This move accelerates practical application and further research. By systematically solving the robustness issue, the team has moved AVTSE technology from a controlled experiment to a viable solution for unpredictable real-world environments where perfect data streams are a luxury.