Llama.cpp b10011 revamps prompt cache and checkpoint management

Llama.cpp, the open-source C++ library for running large language models locally on consumer hardware, has released version b10011. This release focuses on improving the server mode's prompt caching infrastructure. Key changes include a refactoring of prompt cache state ownership to better manage memory, clearing checkpoints when the prompt is cleared to prevent memory bloat, and moving the prompt state data into the server_prompt_cache for cleaner separation of concerns. These optimizations are particularly valuable for developers running multi-turn conversations or batch inference in server deployments.

The release is available as pre-built binaries across many platforms: macOS (Apple Silicon and Intel, with optional KleidiAI acceleration), Linux (x64, arm64, s390x, with Vulkan, ROCm 7.2, OpenVINO, SYCL FP32/FP16), Windows (x64, arm64, with CUDA 12/13, Vulkan, OpenCL Adreno, SYCL, HIP), and Android (arm64). The project continues to be a community favorite with over 120,000 stars on GitHub, enabling efficient local LLM inference without expensive cloud GPUs.