llama.cpp b9659 fixes critical token counting bug across all platforms
The popular local LLM runner patches a miscount in n_tokens affecting inference accuracy.
The latest release of llama.cpp, tagged b9659 and published June 15 by GitHub Actions, addresses a subtle but important bug in the `mtmd` module: a miscount of `n_tokens`. This counter tracks the number of tokens processed or generated during inference, and an off-by-one or similar miscount can lead to incorrect prompt handling, truncated outputs, or misalignment in model responses. The fix ensures that token counters correctly reflect the actual sequence length, which is critical for both single-turn queries and multi-turn conversations.
This release continues llama.cpp's tradition of broad platform support. Binaries are provided for macOS (Apple Silicon with and without KleidiAI, plus Intel x64), Linux (x64 and ARM64 CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL), Windows (x64 and ARM64 CPU, CUDA 12.4 and 13.3, Vulkan, HIP), Android (ARM64 CPU), and iOS as an XCFramework. Special builds are also available for openEuler with ACL Graph acceleration. While this is a minor patch, it underscores the project's commitment to production-grade reliability for local LLM deployment.
- Fixes `mtmd` miscounting of `n_tokens` (affects token counting accuracy)
- Available for macOS Apple Silicon/Intel, Linux (CPU/Vulkan/ROCm/OpenVINO/SYCL), Windows (CPU/CUDA/Vulkan/HIP), Android ARM64, iOS
- Includes builds for openEuler with ACL Graph acceleration; all users on previous releases should upgrade
Why It Matters
Accurate token counting is essential for reliable LLM inference in local, privacy-preserving applications.