MultiMDM enables high-quality few-step generation with multi-mask diffusion
New multi-mask approach drafts and refines tokens for fast, accurate text generation.
Masked diffusion models (MDMs) suffer from collapsed forward trajectories that limit few-step generation. MultiMDM preserves a masking structure: the forward process pushes each clean token toward a designated mask before mixing over the mask set, enabling a backward process that drafts a mask then refines to a clean token. It derives a closed-form ELBO training objective that supports continual training from pretrained MDMs, and formulates a discrete-state consistency distillation with shared-Gumbel coupling. Experiments on pretraining and distillation show MultiMDM provides an effective foundation for principled few-step generation.
- Uses multiple designated masks instead of one, enabling a draft-then-refine generation process.
- Derives a closed-form ELBO that allows continued training from existing pretrained MDMs.
- Introduces discrete-state consistency distillation with shared-Gumbel coupling to reduce pathwise entropy for few-step inference.
Why It Matters
MultiMDM cuts text generation steps significantly, slashing inference costs for real-time applications without quality loss.