llama.cpp b10081 fixes speculative decoding with sidecar models
New update ensures draft models are correctly resolved from sidecar repos.
ggml-org's llama.cpp, the popular C/C++ implementation of LLaMA for efficient local inference, has shipped version b10081. The release addresses a critical issue in speculative decoding—a technique that uses a smaller ‘draft’ model to generate candidate tokens quickly, then verifies them with a larger main model. Previously, when using the `-hfd` flag to point to a Hugging Face repo that contains speculative sidecar models, `llama.cpp` would incorrectly resolve the draft model to the main model of that repo, because `find_best_model()` ignored sidecar files. The explicit draft plan also suppressed sidecar discovery on the HF repo.
With this fix, the draft plan now properly discovers its sidecars and wires them as the draft model, following the fallback pattern of the main plan. This means that a command like `llama-server -hf repo -hfd repo --spec-type draft-dflash` will correctly use the sidecar model as the draft for speculative decoding. The update is available for multiple platforms including macOS ARM64 (with KleidiAI), Linux, Windows (CUDA, Vulkan, OpenVINO, SYCL, HIP), Android, and iOS. This patch streamlines the workflow for developers and researchers using speculative decoding with custom model repos, reducing latency and improving throughput in local inference pipelines.
- Fixes issue where speculative sidecar models from -hfd repos were incorrectly resolved to the main model.
- Now correctly wires discovered sidecars as the draft model, enabling efficient speculative decoding.
- Available for all major platforms including macOS, Linux, Windows, Android, and iOS with various backends.
Why It Matters
Enables reliable, low-latency speculative decoding in llama.cpp, improving inference speed for local AI applications.