CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language

A research team from Technion and Google has introduced CRoCoDiL, a new framework that fundamentally rethinks how diffusion models generate text. Current Masked Diffusion Models (MDMs) operate directly on discrete tokens, which can lead to struggles with long-range dependencies and semantic coherence. CRoCoDiL solves this by shifting the core diffusion process into a continuous, sentence-level semantic space. The method jointly trains an encoder and a demasker, grounding the discrete token demasking of an MDM within smooth, learned latent representations. This creates a novel autoencoder where the decoding step is performed by a diffusion algorithm.

Building on this unified framework, the team introduced two specific algorithms for unconditional text synthesis. The first, Continuous-Then-Discrete (ConThenDisc), is a hybrid approach that generates a full sequence's latent representation in the continuous space before decoding it to tokens in a single step. The second, Continuous-Within-Discrete (ConWithinDisc), is a multi-diffusion strategy that iteratively refines the latent representation throughout the discrete token sampling process. When tested, these methods demonstrated not only superior generation quality but also a dramatic efficiency gain, achieving more than a 10x speedup in sampling compared to prior diffusion-based text models. This represents a significant step toward making diffusion models a practical and high-quality alternative to autoregressive models like GPT for text generation.