EvoSpec dynamically adapts LLM decoding for 1.13x speedup with 27% less memory

Speculative decoding accelerates large language model inference by having a smaller draft model generate tokens that a target model verifies in parallel. However, as vocabulary sizes grow, the output projection layer becomes a bottleneck. Existing static pruning methods reduce this overhead but suffer from severe drops in acceptance rate when handling specialized domains or topic shifts, because they cannot adapt to dynamic distribution changes.

EvoSpec introduces a context-aware mechanism that retrieves critical long-tail tokens through efficient semantic and statistical indexing, then uses a lightweight online alignment strategy with curriculum learning to minimize the distributional gap between draft and target models in real time. Evaluated on EAGLE-3 across coding, law, and medical datasets, EvoSpec achieves a 1.13x speedup over the state-of-the-art static baseline FR-Spec while reducing memory overhead by 27% compared to standard online adaptation methods.