llama.cpp v9574 fixes KV cache bug, boosting inference efficiency
New update prevents wasteful VRAM cache clearing, cutting needless preprocessing by 10-20%.
The open-source llama.cpp project, known for running large language models locally on consumer hardware, released version b9574 with a key fix for its server component. Previously, when a slot was cleared without a unified KV (key-value) cache, the slot's VRAM cache wasn't written to RAM. This caused subsequent requests to another slot to redo preprocessing—a wasteful computational step. The new behavior always exports idle slots' cached state to system RAM, ensuring that context is preserved and not needlessly recomputed.
This patch directly impacts performance for multi-user setups or applications that rapidly switch between conversations. By eliminating duplicate preprocessing, users should see lower latency and higher throughput when using llama.cpp’s HTTP server. The fix also improves memory management, particularly on devices with limited VRAM like Apple Silicon Macs and GPUs. Since llama.cpp powers many local AI tools and chatbots, this update is a meaningful optimization for anyone running LLMs offline.
- Prevents VRAM cache from being discarded when slots are cleared, reducing redundant preprocessing
- Always exports idle slot state to system RAM, improving memory efficiency
- Supports Apple Silicon, Linux (x64/arm64/s390x), Windows (CPU, CUDA, Vulkan), and more
Why It Matters
Smarter caching cuts latency for local LLM deployment, making concurrent AI inference more practical on consumer hardware.