CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

A research team led by Junzhe Shen has published a breakthrough paper titled 'CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think' on arXiv. The work tackles a persistent puzzle in AI: why continuous diffusion language models (DLMs), which have appealing generative dynamics, have consistently underperformed compared to discrete diffusion approaches. Through a controlled token-recovery study, the researchers pinpointed the final 'token rounding' step—the projection from denoised embeddings back to discrete tokens—as the primary performance bottleneck. This insight challenges previous assumptions and provides a clear target for architectural improvement.

Building on this diagnosis, the team proposed CoDAR, a novel two-stage framework. The first stage keeps the diffusion process entirely continuous within a semantic embedding space. The second stage introduces a learned, context-conditional discretizer: an autoregressive Transformer decoder. This decoder cross-attends to the entire denoised embedding sequence to perform 'contextualized rounding,' intelligently converting the continuous representations into a coherent sequence of tokens. Experiments on standard benchmarks like LM1B and OpenWebText show CoDAR substantially closes the quality gap with state-of-the-art discrete DLMs. Crucially, the architecture exposes a simple 'decoder-temperature' hyperparameter, giving practitioners a direct knob to navigate the classic trade-off between output fluency and creative diversity, a level of control often missing in other models.