Llama.cpp's New Update Cuts Token Sorting by 12x — But the Real Win Is What It Unlocks

The open-source llama.cpp project, a leading C/C++ implementation for running large language models locally, has shipped version b9731 with a targeted performance improvement. The update optimizes the get_token_probabilities function, which is called when the server needs to return logprobs for each token position. Previously, the code performed a full sort of the logit vector (typically 128,000 entries) to find the top-n probabilities. Now it uses std::partial_sort, which only orders the top-n items and leaves the rest unsorted. Benchmark results are striking: for a vocabulary of 128,000 tokens and n_top=0 (requesting no additional top probabilities), the full sort took 8,555.6 microseconds per iteration, while the partial sort completed in just 704.3 microseconds—a 12x improvement.

The patch was contributed by Adrien Gallouët of Hugging Face, highlighting the collaborative nature of the open-source AI ecosystem. The change is particularly valuable for applications that frequently query logprobs, such as beam search, reward model scoring, or debugging token probabilities. Given that llama.cpp powers many desktop and server-side LLM deployments, this optimization reduces overhead for users who rely on logprobs in their inference pipelines. The release also includes updated binary builds for macOS (Apple Silicon and Intel), Linux (x64, arm64, s390x with various backends like Vulkan, ROCm, OpenVINO, SYCL), Windows (CPU, CUDA, Vulkan, OpenVINO, SYCL, HIP), and Android arm64.