SelfTTS: cross-speaker style transfer through explicit embedding disentanglement and self-refinement using self-augmentation

A research team led by Lucas H. Ueda has unveiled SelfTTS, a novel text-to-speech architecture designed to transfer emotional speaking styles between different voices. The core innovation is its ability to perform this 'cross-speaker style transfer' without relying on any external, pre-trained models for speaker or emotion recognition. Instead, SelfTTS internally disentangles the two components—who is speaking and how they are speaking—using a combination of Gradient Reversal Layers (GRL) and a cosine similarity loss function. This forces the model to learn separate, clean representations for speaker identity and emotional content directly from the data.

To further refine these representations, the team introduced Multi Positive Contrastive Learning (MPCL), which clusters embeddings based on their speaker and emotion labels, enhancing the model's discrimination power. Perhaps the most clever aspect is the self-refinement stage, where the model uses its own voice conversion capabilities to generate augmented training data, effectively teaching itself to produce more natural-sounding speech. Submitted to Interspeech 2026, the paper reports that SelfTTS achieves superior scores in emotional naturalness (eMOS) and demonstrates more robust stability in preserving both the target speaker's timbre and the intended emotion compared to existing state-of-the-art methods.