TTCD: New diffusion LM uses per-token timing for faster, better generation
A 160M parameter model that surpasses discrete diffusion at high speedups by treating each token's noise-to-clean journey individually.
A new paper from researchers Parikshit Bansal and Sujay Sanghavi presents Token Time Continuous Diffusion (TTCD), a diffusion language model that rethinks how noise is removed during generation. Traditional discrete-space diffusion models suffer inaccuracies when forced to generate multiple tokens in parallel for speed. TTCD operates in continuous space, allowing it to deterministically map Gaussian noise to a complete token sequence without additional sampling noise.
The key innovation is per-token timing: each token can progress from noise to clean at its own rate. Confident tokens speed ahead while uncertain ones take more refinement steps. This not only produces better conditional generation—outperforming discrete models of similar size at high speedups—but also enables differentiated inter-token influences during the refinement process. The team trained a 160M-parameter TTCD model on OpenWebText and then self-distilled it, achieving comparable unconditional quality and superior conditional quality. They also demonstrated gains in structured tasks like Sudoku solving.
- TTCD uses continuous space modeling to avoid errors from parallel token sampling in discrete diffusion models.
- Per-token times allow each token to denoise at its own rate, improving conditional generation quality at high speedups.
- A 160M-parameter TTCD model trained on OpenWebText and self-distilled beats discrete baselines in conditional tasks and Sudoku solving.
Why It Matters
TTCD could accelerate high-quality text generation and structured reasoning while reducing inference artifacts from discrete diffusion.