Parallel Prefix Verification for Speculative Generation

Existing speculative decoding methods accelerate LLM inference by generating draft tokens from a smaller model and then verifying them with the target model. However, these methods are limited by token-level equivalence checks: each token must be verified sequentially, which restricts acceptance length and yields modest speedups. While shifting to semantic-level verification (checking whole phrases or segments) could improve granularity, prior approaches required sequential segment-by-segment checks, introducing overhead that eroded practical gains.

PARSE breaks this limitation with a technique called parallel prefix verification. Instead of iterating over segments, the target model evaluates correctness across multiple prefixes simultaneously in a single forward pass, using a custom attention mask to directly identify the longest valid prefix. This eliminates sequential overhead while making verification compute-efficient. The method is orthogonal to token-level speculative decoding and can be composed with systems like EAGLE-3 for additional speed. Experiments show PARSE achieving 1.25x–4.3x throughput gains over the target model alone and 1.6x–4.5x when combined with EAGLE-3, across diverse models and benchmarks, with negligible accuracy degradation. The paper is available on arXiv (2605.04263) and offers a simple yet powerful way to accelerate LLM inference at scale.