Dustin framework achieves 9.17x end-to-end speedup for long-context LLM generation

Researchers from multiple institutions have introduced Dustin, a draft-augmented sparse verification framework designed to overcome the verification bottleneck in speculative decoding for long-context large language models (LLMs). Speculative decoding improves throughput by using a draft model to generate candidate tokens, which the target model then verifies. However, in long-context settings (e.g., 32k tokens), the repeated loading of Key-Value (KV) caches for verification becomes a latency bottleneck. Existing compression approaches either suffer accuracy loss from static eviction due to saliency shifts or introduce high computational overhead from dynamic selection during verification.

Dustin solves this by combining lookahead signals from the draft model with historical attention from the target model to reliably identify critical tokens across multi-step verification windows. A sparse estimation scheme further reduces overhead by scoring importance only on a minimal subset of attention heads. Evaluated on PG-19 and LongBench using Qwen2.5-72B, Dustin delivers a 27.85x speedup in self-attention and a 9.17x end-to-end decoding speedup for 32k sequences, all while maintaining negligible accuracy degradation. The paper has been accepted to ICML 2026, marking a significant step toward efficient long-context LLM inference.