llama.cpp v.b9761: Model downloads now run in background with live progress
Real-time model load tracking via SSE, plus child-process download for smoother server
ggml-org's llama.cpp, the go‑to C/C++ inference engine for local LLMs, shipped version b9761 with two major server improvements. First, model downloads are now handled by a separate child process (commit #24834), preventing the main server thread from blocking during large file transfers. This means users can initiate a download without halting other requests, a common pain point for self‑hosted setups. Second, the server exposes a new /models/sse endpoint that streams real‑time load progress via Server‑Sent Events. Developers can now build UIs or scripts that show exactly how much of a model has been downloaded or loaded, making large‑model deployments more transparent.
The release also includes a shorter MODEL_DOWNLOAD_TIMEOUT for tests, cleanup of unused code, and fixes for test cases. Builds cover the usual extensive matrix: macOS Apple Silicon (arm64, with KleidiAI optional), Intel x64, iOS XCFramework; Linux on x64 and arm64 with CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL; Windows on x64 and arm64 (CPU, CUDA with multiple DLL versions, Vulkan, OpenVINO, SYCL, HIP); and Android arm64. OpenEuler builds are currently disabled. With 118k stars and 19.8k forks, llama.cpp remains the most popular open‑source LLM runtime, and this update makes it more robust for production server deployments.
- Model downloading now runs in a dedicated child process to avoid blocking the server (commit #24834)
- New /models/sse endpoint provides real‑time load progress via Server‑Sent Events
- Supports 25+ platform/backend combinations including CUDA, ROCm, Vulkan, SYCL, OpenVINO, and HIP
Why It Matters
Self‑hosting LLMs becomes more reliable with background downloads and live progress, enabling smoother API operations.