llama.cpp b10313 adds LRU scheduler for faster inference
New LRU router in llama.cpp reduces queue waits with handle coalescing...
The ggml-org team has shipped llama.cpp b10313, a new release of the widely adopted C++ inference engine for running large language models locally. The headline feature is a new LRU (least-recently-used) scheduler added to the server router via pull request #26572. This scheduler introduces lru_sched handle coalescing, a mechanism that optimizes how requests are batched and scheduled, allowing requests to leave the waiting queue more efficiently. The change also includes dedicated tests and a fix for streaming cases, ensuring that the router handles long-running token generation more reliably.
llama.cpp b10313 also reinforces its cross-platform footprint, with prebuilt binaries available for macOS (Apple Silicon, Intel, and KleidiAI-enabled ARM builds), iOS, Linux (CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL), Android (CPU and OpenCL Adreno), and Windows (CPU, CUDA 12 and 13, Vulkan, OpenVINO, SYCL, HIP). The release is part of the project's rapid iteration cycle, and the LRU scheduler is aimed at improving throughput for concurrent users when hosting LLM inference on a single machine via the llama.cpp server.
- New LRU scheduler in server router with lru_sched handle coalescing
- Requests can now leave the waiting queue faster, improving concurrency
- Includes tests and a stream-case fix, plus builds for CPU, CUDA, Vulkan, ROCm, and more
Why It Matters
Better request scheduling in llama.cpp means smoother, faster local LLM serving for developers and self-hosted inference.