**Introducing SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding**

NVIDIA researchers have launched SPEED-Bench, a comprehensive new benchmark designed to solve the fragmented and often unrealistic evaluation of Speculative Decoding (SD). SD is a critical technique where a small 'draft' model predicts several tokens ahead of a larger target LLM, allowing for parallel verification and significant inference speedups—often 2-3x faster—without changing the output. Current benchmarks fail to reflect real-world conditions, using small datasets, short sequences, or single-batch processing. SPEED-Bench provides a unified ecosystem to standardize testing across the field.

The benchmark consists of two core components. First, a 'Qualitative' data split uses a custom algorithm on prompt embeddings to maximize semantic diversity across categories like conversation, translation, and reasoning, allowing for accurate measurement of draft model acceptance rates. Second, a 'Throughput' split aggregates data into fixed Input Sequence Length buckets (1k to 32k tokens) and supports high-concurrency batch sizes up to 512, enabling evaluation of true system-level speedups under production loads. This dual approach captures both the algorithmic quality and the practical hardware performance of SD techniques.

By integrating with production-grade inference engines, SPEED-Bench offers a realistic measurement framework that reveals performance characteristics often hidden by simpler tests. This allows AI engineers to make informed decisions when selecting draft models like smaller Llama or Phi variants and tuning SD algorithms for specific deployment scenarios, ultimately leading to more efficient and cost-effective LLM serving.