Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes

A research team led by Fangyu Ding has introduced Deletion-Insertion Diffusion (DID) language models, a significant architectural shift for diffusion-based AI. Accepted at the prestigious ICLR 2026 conference, DID moves beyond the established Masked Diffusion Language Model (MDLM) paradigm, which relies on masking and unmasking tokens. Instead, DID rigorously formulates text generation as a discrete diffusion process of token deletion and insertion. This core change directly targets two major sources of computational overhead in MDLMs: the wasted processing on placeholder <MASK> tokens and the padding (<PAD>) tokens required for fixed-length training.

By eliminating these inefficiencies, DID achieves notable gains in both training and inference speed while improving modeling performance. Crucially, the architecture natively supports variable-length sequences without artificial padding, offering greater flexibility for real-world text generation tasks. Furthermore, the insertion process introduces an intrinsic self-correction mechanism, allowing the model to dynamically adjust token positions during generation for higher-quality output. The team developed a novel score-based training approach involving subsequence counting, solved efficiently via a parallelized dynamic programming algorithm. Experiments show DID outperforms baseline MDLMs and other insertion-based models in speed and quality without extensive hyperparameter tuning.