KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

A team of researchers has introduced KnapSpec, a novel framework for accelerating large language model (LLM) inference through a more intelligent form of self-speculative decoding (SSD). Unlike previous SSD methods that use static heuristics to skip layers and create a faster draft model, KnapSpec dynamically selects which layers to skip by treating the problem as a classic knapsack optimization. The goal is to maximize the overall tokens-per-time throughput, accounting for the fact that computational bottlenecks—particularly in Attention layers—shift dramatically with longer context lengths. This training-free, plug-and-play approach ensures the original model's output distribution is preserved while seeking the fastest possible inference path.

KnapSpec's key innovation is decoupling the latency modeling of Attention and MLP layers, creating hardware-aware cost functions that depend on context length. This allows its parallel dynamic programming algorithm to identify the optimal draft configuration in real-time. The researchers also provided the first rigorous theoretical analysis, establishing cosine similarity between hidden states as a mathematically sound proxy for token acceptance rate, which underpins the method's drafting faithfulness. In experiments on models like Qwen3 and Llama3, KnapSpec consistently outperformed existing SSD baselines, achieving speedups of up to 1.47x. This represents a significant step towards practical, high-speed inference for long-sequence tasks without the cost of model retraining.