llama.cpp b9978: Checkpoint eviction optimizes LLM server memory
New release cuts memory waste by evicting stale checkpoints during inference.
Deep Dive
ggml-org released llama.cpp b9978 on July 12, featuring a server change: evict checkpoints within min-step of each other (#25472). Builds are available for macOS (Apple Silicon and Intel, with some disabled), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL, plus s390x), Android (arm64), Windows (CPU, arm64, CUDA 12 & 13, Vulkan, OpenCL Adreno, OpenVINO, SYCL, HIP), and openEuler (x86 and aarch64, some disabled). The release also includes UI assets. No other details or claims are provided.
Key Points
- Server now evicts checkpoints that are within min-step of each other, reducing memory usage during inference.
- Release b9978 covers 20+ build targets including macOS, Linux, Windows, Android, iOS, and specialized backends like ROCm and Vulkan.
- Fix addresses a specific GitHub issue (#25472), improving stability for long-running local LLM servers.
Why It Matters
Reduces memory overhead for self-hosted LLM servers, enabling cheaper and more reliable local AI deployment.