llama.cpp b10429 adds live /metrics and /slots during decode
Llama.cpp's new build lets you monitor inference in real time, no blocking.
llama.cpp, the wildly popular open-source LLM inference project from ggml-org, shipped release b10429 on August 14. The headline change: the server now permits /metrics and /slots requests during llama_decode(), meaning operators can monitor active inference slots and performance metrics concurrently with ongoing generation. Previously, these endpoints would block or queue until decode completed, making real-time monitoring of production workloads difficult. This update also cleans up server queue internals and handles process_mtmd_chunk more efficiently.
The release covers an extensive matrix of builds—macOS (Apple Silicon incl. KleidiAI), Linux (CPU, Vulkan, ROCm 7.14, OpenVINO, SYCL), Windows (CUDA 12/13, Vulkan, OpenCL Adreno), Android arm64, and even openEuler with Ascend NPUs. With 124k stars and 21.7k forks, llama.cpp remains the go-to for local LLM deployment. For developers running concurrent requests, this is a quality-of-life upgrade: you can now track slot availability and token metrics in real time, enabling better load balancing and autoscaling decisions during inference.
- Access /metrics and /slots endpoints during llama_decode() for live monitoring
- Release b10429 from ggml-org includes server queue cleanup and mtmd_chunk handling fixes
- Supports 20+ build targets including CUDA 12/13, Vulkan, ROCm, and Apple Silicon with KleidiAI
Why It Matters
Real-time inference observability helps operators optimize LLM serving, reducing latency and improving resource utilization in production.