Researcher extracts voice styles from frozen TTS models

Researcher Gyeongmin Kim has demonstrated a novel method to extract voice styles from proprietary or 'frozen' text-to-speech (TTS) systems without access to original model weights or reference encoders. Published on arXiv under *Extracting Voice Styles from Frozen TTS Models via Gradient-Based Inverse Optimization*, the paper introduces a gradient descent-based technique that inverts the TTS synthesis pipeline to directly optimize a style vector from raw audio input.

The approach targets TTS systems that ship with preset style vectors but omit the original reference encoder used to generate them. By optimizing only the style vector against time-pooled WavLM embeddings—discarding temporal alignment—Kim’s method bypasses the need for transcripts or phonetic alignment. On a benchmark of 154 speakers from two corpora, speaker similarity scores dramatically improved: ECAPA-TDNN similarity rose from 0.132 to 0.413, and ResNet similarity from 0.099 to 0.401. A verification system accepted 53% of recovered voices as the target at equal-error rate, compared to just 1% for the original presets. This breakthrough enables voice cloning from limited audio data, raising both creative potential and ethical concerns in synthetic speech.