New AI Uses Your Lips to Silence Everyone Else on Calls

Picture a video call from a coffee shop: someone at the next table is talking, a blender is running, and your colleagues struggle to hear you. A new research paper describes an AI that fixes this by combining two senses — hearing and sight. It watches the movement of your lips on camera while listening to the audio, then keeps only your voice and throws away everything else. Because it works with a delay of just 20 milliseconds — far faster than the human ear can notice — it can run live during a call rather than being cleaned up afterward.

The starting point was an AI that already knew what your voice sounds like, having been given a sample of you speaking in advance. That system produced beautiful-sounding audio, but it still grabbed the wrong person's voice in 46% of two-person recordings — nearly half the time. The fix was surprisingly simple: give the AI information about mouth movements as well. That one addition, plus some joint training, cut the error rate to 1.6%. That is the difference between a tool you'd never trust and one you'd actually use.

The researchers also tested how it sounds to real listeners, not just how well it separated voices. On two sets of recorded meetings, people rated the new system noticeably better than an existing audio-visual competitor. It also behaves politely: when only you are speaking, it leaves your voice untouched, and when a competing speaker is present but you are silent, it mutes them. Importantly, it kept its edge even in recordings with more speakers than it was trained on.

One honest caveat: this is a research paper, not a product you can download today. It also needs a camera pointed at your face, and a sample of your voice recorded beforehand. But the ingredients — a webcam, a microphone, and a modest AI model — are already in every laptop and phone, so the leap from paper to feature could be short.