llama.cpp b9550 fixes KV cache overflow bug in speculative decoding
A critical fix prevents crashes when target context is smaller than draft default.
The latest release of llama.cpp, version b9550, addresses a critical bug in the key-value (KV) cache sharing mechanism used during speculative decoding. The issue occurred when a fitted target context ended up smaller than the draft model's default context size. In such cases, the assistant's view of the KV cache would be oversized, overflowing the shared tensors and causing a ggml_view_4d size assertion failure during graph reservation. The patch, contributed to #24267, now ensures that when cells are shared, the assistant's view follows the source cache size rather than a fixed default, preventing memory overruns and crashes.
This fix is particularly important for developers running speculative decoding across different model sizes or quantizations, where context length mismatches are common. The release includes prebuilt binaries for a wide range of platforms: macOS (Apple Silicon with and without KleidiAI, Intel), iOS as an XCFramework, Linux on multiple architectures (x64, arm64, s390x) with backends including Vulkan, ROCm 7.2, OpenVINO, and SYCL, Android arm64, Windows (x64 and arm64) with CPU, CUDA 12/13, Vulkan, and HIP support, plus openEuler with Ascend NPUs. Users should update to avoid silent errors or crashes during inference.
- Fixes KV cache overflow when target context smaller than draft default in speculative decoding
- Prevents ggml_view_4d size assertion failure during graph reservation
- Available for macOS, iOS, Linux, Android, Windows, and openEuler with multiple GPU backends
Why It Matters
This fix stabilizes speculative decoding in llama.cpp, enabling faster inference without crashes on mixed-context workloads.