b8876

The open-source powerhouse behind efficient local AI inference, ggml-org, has shipped a targeted but important update to its llama.cpp project. Version b8876 specifically addresses a bug (#22168) in the speculative decoding feature, a performance optimization technique where a smaller, faster model drafts tokens that a larger model then verifies. The bug occurred during 'low acceptance streaks'—when the larger model repeatedly rejected the draft tokens. The `i_last` variable, which tracks position in the context, wasn't being reset to zero, causing the system to rebuild its speculative map from an incorrect starting point. This could lead to less efficient inference and potential context confusion.

The fix is deceptively simple: reset `i_last` to zero when a low acceptance streak is detected. This ensures the current, correct context is always used when rebuilding the speculative decoding map, leading to more stable and performant text generation. The release is notable for its extensive cross-platform support, with GitHub Actions automatically building and distributing binaries for 28 distinct targets. These range from standard CPU builds for Windows, Linux, and macOS (including Apple Silicon) to specialized versions leveraging Vulkan, CUDA, ROCm, SYCL, HIP, and OpenVINO for accelerated performance, plus builds for mobile (iOS, Android) and the openEuler OS.