b8786

The open-source project llama.cpp, maintained by ggml-org, has released a crucial performance fix in commit b8786. This update resolves a significant 30% speed regression that was plaguing several advanced "reasoning" language models, including Google's Gemma 4, Kimi's K2, and Mistral's Ministral 3. The bug was introduced when support for thinking tags (like `thinking_start_tag`) was added, which inadvertently forced the creation of a "reasoning budget sampler" even when no token budget was set. This sampler, while idle, had the critical side effect of disabling backend sampling—a key optimization where the GPU directly selects the next token, avoiding costly data transfers to the CPU.

The fix is elegantly simple: it now checks if a reasoning budget is explicitly configured (e.g., 0, 128, or 1024 tokens) before creating the sampler. If the budget is unlimited (the default -1 value), the sampler is skipped entirely, allowing backend GPU sampling to remain active. This restores the original inference speeds, such as the reported case where performance jumped back from 70 tokens/second to 98 tokens/second on Vulkan. The commit also preserves the sampler when using "grammar_lazy" mode to ensure tool-calling functionality remains intact. This update is vital for developers and users running these state-of-the-art reasoning models locally, ensuring they get the full computational efficiency llama.cpp is known for.