Llama.cpp now with a true reasoning budget!
New feature lets developers limit AI 'thinking' tokens, boosting efficiency by preventing endless reasoning loops.
The popular open-source inference engine llama.cpp has rolled out a significant upgrade: genuine reasoning budget control. Previously, the `--reasoning-budget` parameter was essentially non-functional, serving only to disable thinking entirely. The new implementation uses the sampler mechanism to actively count tokens during a model's reasoning phase and force-terminate it when a user-defined limit is reached. This provides developers with a crucial tool to manage computational cost and latency, especially for models like StepFun 3.5 that are designed to 'think' extensively by default.
Initial testing revealed a challenge: abruptly cutting off reasoning degraded performance. On the Qwen3 9B model, enforcing a budget without warning crashed its HumanEval score from 94% to 78%. To solve this, the team added a companion `--reasoning-budget-message` flag. This inserts a gentle prompt (e.g., '... thinking budget exceeded, let's answer now.') before the cutoff, allowing the model to gracefully conclude its thought process. With this message, the Qwen3 9B's score recovered to 89%. The feature also allows users to forcibly restrict strongly 'thinking' models, though setting the budget to zero can cause erratic behavior as the model struggles with its constrained design.
This update transforms reasoning from an open-ended process into a tunable resource. Developers can now experiment with budget sizes and transition messages to find the optimal balance between answer quality, token usage, and speed for their specific application and model.
- The new sampler-based `--reasoning-budget` flag actively limits the number of tokens a model uses during its internal 'thinking' phase.
- A `--reasoning-budget-message` flag mitigates performance drops by prompting the model before cutoff, recovering Qwen3 9B's HumanEval score from 78% to 89%.
- Enables cost/performance tuning for 'thinking' models like StepFun 3.5, though forcing zero reasoning can cause unstable behavior.
Why It Matters
Gives developers fine-grained control over AI inference cost and latency, making advanced reasoning models more predictable and economical to deploy.