DSA-Tokenizer: New speech tokenizer disentangles meaning from voice with flow matching

A team of researchers has introduced DSA-Tokenizer, a novel speech tokenization method that cleanly separates semantic content (what is said) from acoustic style (who said it and how). Unlike prior tokenizers that fuse these aspects or achieve only partial disentanglement, DSA-Tokenizer uses distinct optimization constraints: semantic tokens are supervised by ASR to capture linguistic information, while acoustic tokens are optimized for mel-spectrogram reconstruction to encode pitch, timbre, and prosody.

To enable both faithful reconstruction and controllable generation, the architecture includes a hierarchical Flow Matching decoder and a joint training strategy that combines reconstruction with context inpainting. This allows zero-shot voice cloning across different utterances. The authors also distill the DiT backbone and apply GAN fine-tuning to reduce inference to just 4 sampling steps while improving synthesis quality. Experiments report strong disentanglement metrics, low word/character error rates, and efficient high-fidelity generation. The paper suggests that such disentangled tokenization offers a more effective interface for downstream large-model speech generation tasks.