GRAFT method gives TTS per-word pronunciation control from short audio samples

A team of researchers has published GRAFT, a new technique that gives text-to-speech (TTS) systems precise per-word pronunciation control using a short audio sample of the word. The method, detailed in a paper on arXiv, tackles a persistent problem in zero-shot TTS: mispronunciation of proper nouns, loanwords, and technical terms. While existing systems rely on text or phoneme input, they lack an acoustic handle to ensure the correct sound for each word. GRAFT solves this by grafting the acoustic features of a reference audio clip onto the target output, using the model's own speech tokenizer and voice conversion to separate the hint speaker from the target voice.

GRAFT achieves a 22-39% reduction in phoneme error rate for target words across a five-language objective benchmark, outperforming both text-based and phoneme-conditioned open-source zero-shot TTS systems. In a blind English listening study, human raters consistently ranked GRAFT first, judging its pronunciation of difficult words as closest to a reference recording. The method preserves speaker similarity and overall naturalness, meaning the output retains the desired voice while accurately pronouncing even obscure terms.

The practical implications are significant for professionals relying on TTS for audiobooks, voice assistants, or content localization. GRAFT enables users to provide a quick spoken hint for any problematic word, ensuring correct pronunciation without requiring extensive phonetic transcription or model retraining. By combining voice conversion with per-word conditioning, the system can accept hints from any speaker—even a different gender or accent—and still output the target voice. This flexibility makes GRAFT a promising step toward truly natural and accurate synthetic speech.