llama.cpp b9688 adds model management API and real-time SSE streaming
The open-source LLM runtime now lets you manage and download models via API with live updates.
The latest release of llama.cpp, the widely-used C++ inference engine for large language models, introduces significant server-side improvements. Version b9688 adds a model management API to the router, allowing developers to dynamically load, unload, and switch models without restarting the server. This is paired with Server-Sent Events (SSE) for real-time updates on model status and inference progress. A new download API and delete endpoint simplify pulling models from remote sources directly into the runtime.
The release also includes numerous bug fixes and stability improvements, along with expanded build support for macOS Apple Silicon (with KleidiAI), Intel x64, Linux on various architectures, Windows (CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and Android ARM64. With over 117,000 stars on GitHub, llama.cpp remains the go-to solution for local LLM deployment, and this update makes it more production-ready by offering proper model lifecycle management.
- New model management API in the router server enables dynamic model loading/unloading.
- SSE real-time updates provide live status on inference and model state changes.
- Added download API and delete endpoint for managing remote model files locally.
Why It Matters
Local LLM serving now gains production-grade API control, enabling scalable model swapping and remote model management.