AdaLook boosts diffusion models with adaptive multi-step lookahead decoding
New adaptive lookahead method for masked diffusion models improves accuracy by up to 40%.
Masked diffusion language models (DLMs) generate text in parallel by iteratively refining masked tokens, offering a faster alternative to autoregressive decoding. Recent lookahead-based methods improve the accuracy-efficiency trade-off by exploring future decoding states before updating tokens. However, existing approaches rely on shallow one-step lookahead, which optimizes immediate information gain but is suboptimal for longer decoding trajectories. A naive extension to deeper rollout also fails because fixed-depth lookahead adds unnecessary computation and cannot adapt to heterogeneous intermediate states.
AdaLook solves this by dynamically deciding when to continue rollout based on candidate-score variance, and by enabling branch expansion when intermediate states require deeper exploration. This avoids wasted computation while allowing the decoder to re-trigger lookahead from informative positions. Evaluated on multiple DLM benchmarks, AdaLook achieves a superior accuracy–decoding steps trade-off compared to one-step lookahead methods, demonstrating faster convergence and higher quality generation without additional overhead. The framework is model-agnostic and can be integrated into existing DLM pipelines.
- Adaptive rollout depth based on candidate-score variance reduces unnecessary computation by up to 50%.
- Branch expansion from informative intermediate states improves generation accuracy on long sequences.
- Outperforms one-step lookahead decoding across multiple benchmarks with fewer decoding steps.
Why It Matters
AdaLook enables faster, more accurate parallel text generation, pushing diffusion models closer to real-time NLP applications.