Speaker Disentanglement of Speech Pre-trained Model Based on Interpretability

A team of researchers has published a breakthrough paper addressing a fundamental problem in speech AI: the entanglement of speaker identity and content information in pre-trained models like HuBERT and WavLM. This entanglement creates dual problems—speaker bias that harms content recognition accuracy, and privacy risks where supposedly anonymized speech embeddings can still reveal who's speaking. The researchers' solution comes in two parts: first, they created InterpTRQE-SptME, a novel benchmark that uses SHAP (SHapley Additive exPlanations) interpretability analysis to directly measure exactly how much speaker information remains in content embeddings, moving beyond indirect metrics.

Second, and most practically, they developed InterpTF-SptME, a filtering method that applies these interpretability insights to actively remove speaker-specific information. When tested on the VCTK dataset across seven major speech models, the results were striking. Their SHAP Noise filtering technique reduced residual speaker information from 18.05% to nearly zero percent. Crucially, this disentanglement came with minimal impact on content recognition—the Character Error Rate (CTC loss) increased by less than 1%, meaning the models could still understand what was said while forgetting who said it.

The method's model-agnostic nature and requirement of no retraining make it immediately applicable to existing speech AI systems. This represents a significant advance over previous approaches that either sacrificed too much accuracy or couldn't properly quantify what they were removing. The technique effectively creates a privacy filter for speech embeddings, potentially enabling safer voice data sharing and more robust speech recognition systems that aren't biased by speaker characteristics.