llama.cpp adds model management API for on-demand downloads and lifecycle control

llama.cpp, the popular C++ implementation for running LLMs locally, just merged pull request #23976, introducing a powerful model management capability over its API. Now, instead of manually placing model files or restarting the server, users can dynamically load and unload models from a designated directory—or even download new models on the fly—all through simple API calls. This turns llama.cpp into a self-contained model serving solution where no external orchestration tool is needed.

While there's no user interface yet (though one is promised soon), this update is a game-changer for developers building custom local AI pipelines. It means deploying a single server instance that can swap between different models without downtime, fetch new ones as needed, and expire old ones—all programmatically. For edge deployments, privacy-sensitive applications, or anyone wanting a lightweight alternative to heavy cloud dependencies, this feature drastically simplifies the infrastructure. The API-based lifecycle management brings llama.cpp closer to being a drop-in replacement for hosted model APIs, but entirely on your own hardware.