llama.cpp b9566 fixes SWA-only draft head crash with kq_mask guard
New patch resolves null pointer crash in speculative decoding with sliding window attention
The ggml-org/llama.cpp project released version b9566 with a critical bug fix for speculative decoding. The issue occurred when using a SWA-only (Sliding Window Attention-only) draft head, such as StepFun's Multi-Token Prediction (MTP) architecture. In this configuration, the base sub-cache remains empty, causing the kq_mask buffer to stay null and triggering an assertion error during model loading. The commit, co-authored by Georgi Gerganov, introduces guards for each mask on its own buffer in both set_input and can_reuse functions, covering base and SWA paths.
The release is a stable update with comprehensive platform support: macOS (Apple Silicon and Intel), iOS, Ubuntu (x64/arm64, with Vulkan, ROCm, OpenVINO, SYCL), Android (arm64), Windows (x64/arm64 with CUDA 12/13, Vulkan, HIP), and openEuler. The fix ensures reliable inference for users leveraging advanced speculative decoding techniques that combine draft heads with sliding window attention, a common optimization for long-context LLMs.
- Bug caused null kq_mask buffer when SWA-only draft head left base sub-cache empty
- Fix guards each mask on its own buffer in set_input and can_reuse for both base and SWA paths
- Release b9566 is a stable update with cross-platform builds including CUDA, Vulkan, ROCm, and more
Why It Matters
Keeps local LLM inference reliable for advanced speculative decoding techniques using SWA-only draft heads.