EntMTP: New scheduler speeds up LLM inference by up to 36%
Training-free entropy adaptation boosts multi-token prediction throughput.
Existing multi-token prediction models like Hydra and Medusa use fixed tree-based attention topologies throughout generation, wasting compute on unpredictable (high-entropy) text and under-exploiting predictable (low-entropy) regions. Carrie Chen introduces EntMTP, a training-free scheduler that toggles between a set of task-specific Pareto-optimal tree topologies conditioned on a running estimate of local generation entropy. This aligns speculation depth with actual context predictability, maximizing expected accepted-token throughput without sacrificing quality.
Evaluated on four diverse benchmarks—Humaneval, ShareGPT, GSM8k, and Litbench—EntMTP consistently achieves a 1.15x speedup against Hydra and a peak 1.36x speedup against Medusa. The method requires no retraining, adapts in real time, and uses only 7 pages with 5 figures to describe. It represents a practical, low-overhead approach to making LLM inference faster by exploiting the natural entropy patterns of language.
- Dynamically adjusts speculation depth based on local entropy of generated text
- Achieves 1.15x–1.36x speedup over static tree-based MTP methods like Hydra and Medusa
- Training-free scheduler tested on Humaneval, ShareGPT, GSM8k, and Litbench
Why It Matters
Reduces wasted compute in LLM inference by matching speculation to context predictability, enabling faster generation without quality trade-offs.