Polestar boosts diffusion LLM inference: 3.7x faster, 10.7% more accurate

Diffusion large language models (dLLMs) generate text by iteratively refining random noise, but their bidirectional attention mechanism makes inference painfully slow. Two core bottlenecks: the KV-cache can't be reused efficiently because token representations drift across steps, and existing methods to increase parallelism using static thresholds often degrade output quality. Researchers observed that both problems stem from the same root cause—token drift—and built Polestar to exploit it.

Polestar is a training-free framework with two components. Polestar-Cache monitors drift to identify stale cache positions and performs sparse refreshes, enabling KV-cache reuse without the overhead of full recomputation. Polestar-Commit looks for sharp drift events that signal a token is stable enough to be committed, allowing multiple tokens to be decoded in parallel while preserving coherence. On benchmarks like GSM8K and HumanEval, Polestar delivers up to 10.73% higher accuracy, up to 3.7x throughput improvement, and a decoding parallelism of 3.67 tokens per forward pass—outperforming all prior methods on the accuracy-throughput Pareto frontier.