Emotion vectors in speaker embeddings enable precise TTS emotional control

A new paper on arXiv (2606.05367) from Daniel Oliveira de Brito and Arnaldo Candido Junior tackles emotional expressivity in language-model-based text-to-speech (LM-TTS). The team tested four methods for transferring emotional prosody across speakers—LoRA fine-tuning, continuous codec embeddings, discrete codec tokens, and the speaker embedding (x-vector) from an ECAPA-TDNN encoder. Through systematic elimination on the 1.7B-parameter Qwen3-TTS-12Hz, they found the x-vector carries the dominant emotional signal. This enabled a training-free approach: compute an “emotion direction” by averaging x-vectors from emotional speech and subtracting neutral ones, then add that vector (scaled by α) to a target speaker's neutral x-vector.

Using the ESD corpus (English) as the emotion source and emoUERJ (Brazilian Portuguese) as the cross-lingual target, the method improved emotion2vec cosine similarity by +0.29 over the in-context learning baseline on English held-out speakers and +0.09 on Portuguese speakers—all while maintaining speaker identity (WavLM SECS ≥0.88 for the multi-speaker variant) and intelligibility (WER ≈0). These results suggest that prior failures of centroid-arithmetic style control in token-based TTS can be circumvented by operating on the speaker embedding itself, offering a lightweight, retraining‑free path to emotional expressivity in production TTS systems.