llama.cpp b10413 auto-detects draft model types for speculative decoding
No more --spec-type flags: local draft models now activate speculative decoding automatically
Speculative decoding in llama.cpp just got a significant quality-of-life upgrade. Release b10413, pushed by ggml-org maintainers on August 13, automatically detects the speculative decoding draft model type when loading local GGUF files via the -md flag. Previously, auto-detection only looked at Hugging Face repo sidecars, so local drafts would load into VRAM but speculative decoding would silently stay disabled because the spec type remained NONE.
The fix reads the general.architecture field from the draft model's GGUF header directly. If the architecture is dflash and the model contains a markov_w1.weight tensor, it maps to draft-dspark; if dflash lacks the markov head, it maps to draft-dflash. The detection also emits a LOG_INF message so users can confirm when spec decoding has been enabled. As part of code review, the logic moved into the speculative module as common_speculative_types_from_gguf(), using RAII via gguf_context_ptr and following the project's single-line comment style. The release also documents the single-file assumption, leaving split-GGUF edge cases for a future update.
For practical use, b10413 ships prebuilt binaries for an enormous range of targets: macOS Apple Silicon (with optional KleidiAI), Intel x64, iOS, Linux variants for CUDA 12/13, ROCm, Vulkan, OpenVINO, SYCL, Android arm64, Windows x64, and even openEuler with Ascend ACL. This is a maintenance-level release aimed at correctness rather than new model support, but it eliminates a frustrating setup step for anyone running local draft models for faster token generation.
- Fixes silent failure where local draft models loaded with -md never activated speculative decoding because auto-detection ignored local GGUF files
- Auto-detection reads general.architecture from GGUF header: dflash with markov_w1.weight maps to draft-dspark, plain dflash maps to draft-dflash, with LOG_INF confirmation
- Refactored into speculative.cpp as common_speculative_types_from_gguf() using RAII, and ships binaries for CUDA 12/13, ROCm 7.14, Vulkan, OpenVINO, SYCL, and more
Why It Matters
Simplifies local speculative decoding setup, cutting VRAM overhead while speeding up generation for llama.cpp users.