EmoInstruct-TTS lets you control 48 emotions in AI speech with text instructions

EmoInstruct-TTS tackles a core limitation of current emotional text-to-speech systems: the reliance on coarse emotion labels like 'happy' or 'sad' that fail to capture subtle variations. The researchers developed Emotion2embed, a supervised semantic-acoustic embedding that covers 48 distinct emotional states, each with fine-grained categories and intensity levels. This allows users to specify nuanced emotions through natural language instructions such as 'a gentle, wistful melancholy' rather than selecting from a fixed list. The framework uses a dual-path architecture: one path handles semantic planning via an LLM-based synthesis pipeline, while the second path generates emotion embeddings from the user's instruction. To bridge the gap between natural language instructions and acoustic emotion representations, they designed the Instruction-Conditioned Emotion Flow Model (ICE-Flow), which converts free-form text into acoustically grounded embeddings that integrate seamlessly into the synthesis pipeline.

The integration of ICE-Flow into the LLM-based TTS system enables explicit emotional control while preserving semantic planning and naturalness. Audio demos are available online, and the paper has been submitted to Interspeech 2026. Experiments compare EmoInstruct-TTS against strong baselines, showing significant improvements in both emotional controllability — the ability to precisely match the intended emotion — and overall speech naturalness. The model handles a wide range of emotions from subtle to intense, and the dual-path approach ensures that varying the emotion does not degrade the clarity or prosody of the speech. This work represents a step toward more expressive, user-customizable speech synthesis, with applications in virtual assistants, audiobooks, gaming, and accessibility tools where emotional nuance enhances user experience.