UtterTune uses LoRA for precise Japanese pronunciation control in TTS
Lightweight adaptation lets you tweak pitch accents and phonemes without retraining
UtterTune, proposed by Shuhei Kato in a new arXiv paper (2508.09767), introduces a lightweight adaptation method for multilingual text-to-speech systems built on large language models. It uses low-rank adaptation (LoRA) to edit and control pronunciation at the phoneme level specifically for Japanese—the target language—without needing an explicit grapheme-to-phoneme (G2P) module. This is critical because modern LLM-based TTS models often process minimally encoded text (e.g., byte-pair encoding), making accurate phonetic and prosodic modeling difficult. UtterTune enables control over segmental pronunciation and pitch accent while keeping the model's performance intact for other languages.
The system maintains high naturalness and speaker similarity in a zero-shot setting, meaning it can adapt to new voices without retraining. The LoRA approach is parameter-efficient, adding only small low-rank matrices to existing weights. Objective metrics and subjective listening tests confirm that UtterTune preserves overall speech quality while granting precise phonetic control. The paper also clarifies its precedence over a subsequent technical report on token-based pronunciation control, and provides code, training data, LoRA weights, and audio samples for reproducibility.
- Uses LoRA (low-rank adaptation) to avoid full model retraining, adding only small matrices to existing weights
- Enables phoneme-level control of pronunciation and pitch accent for Japanese in multilingual LLM-based TTS
- Maintains naturalness and zero-shot speaker similarity across languages, validated by objective and subjective tests
Why It Matters
UtterTune offers a practical, lightweight way to fix mispronunciations in multilingual TTS without sacrificing quality or language coverage.