b8982
The popular open-source LLM engine patches critical vocabulary mismatch detection in speculative sampling.
ggml-org's llama.cpp, the ubiquitous open-source C/C++ inference engine for large language models, shipped version b8982 on April 30. This release is primarily a bugfix that resolves vocabulary compatibility checks in the speculative decoding example (speculative/speculative.cpp). The fix ports logic from PR #22358, ensuring that when the draft model and target model have different vocabularies, the system now correctly logs using 'vocab_tgt' and 'vocab_dft' variables instead of incorrectly referring to context objects. The patch was contributed by Petros Sideris (Nokia).
Speculative decoding is a technique that uses a smaller, faster 'draft' model to generate multiple tokens in parallel, then validates them with a larger target model—boosting throughput without quality loss. Correct vocabulary handling is essential because mismatches can cause silent errors. b8982 also continues llama.cpp's tradition of extensive cross-platform support: ready-to-use binaries are available for macOS (Apple Silicon, Intel, iOS), Linux (x64, arm64, s390x, with Vulkan, ROCm 7.2, OpenVINO, and SYCL backends), Android (arm64), Windows (x64 and arm64 with CPU, CUDA 12/13, Vulkan, SYCL, and HIP), and openEuler (x86 and aarch64 with ACL Graph optimizations). The project remains one of the most starred on GitHub (108k stars, 17.6k forks), reflecting its critical role in the open-source AI ecosystem.
- llama.cpp b8982 fixes vocabulary compatibility checks in the speculative decoding example (speculative/speculative.cpp)
- Patch ports logic from PR #22358 to correctly log mismatches between draft and target model vocabularies
- Builds available for 20+ platform/backend combinations including Apple Silicon, CUDA 12/13, ROCm 7.2, Vulkan, SYCL, and multiple OS architectures
Why It Matters
Speculative decoding boosts LLM inference speed by 2-3x; fixing vocab checks prevents silent errors in production deployments.