SENSE boosts LLM inference 3.26x with semantic embedding navigation

Speculative decoding (SD) accelerates LLM inference by using a lightweight draft model to propose tokens verified in parallel by the target model. Retrieval-based SD (RSD) is popular for being plug-and-play, but it suffers from brittle lexical dependencies that make both retrieval and verification fragile when surface forms vary. To overcome this, researchers from the paper present SENSE (Semantic Embedding Navigation with Soft-gated Evaluation). Instead of matching tokens lexically, SENSE anchors retrieval on the hidden states of the target model, establishing robust semantic alignment. A soft-gated evaluation module then validates semantic equivalence rather than exact string matches, allowing the system to accept paraphrases and semantically similar candidates.

Extensive experiments across diverse domains show SENSE outperforming multiple baselines on the LLaMA and Qwen model families. It achieves up to 4.09 mean acceptance length and a 3.26x end-to-end speedup, all while maintaining the same generation quality as standard autoregressive decoding. The method is a drop-in replacement for existing retrieval-based SD approaches, requiring no retraining of the target model. The authors also decompose existing methods into atomic primitives for granular comparison, and plan to release code upon publication. This work makes speculative decoding significantly more practical for production environments where latency and cost matter.