llama.cpp b10199 adds input embeddings to server for token generation
The latest llama.cpp release lets server use inp embd for next-token prediction
The ggml-org team released llama.cpp b10199, a minor but impactful update to the popular local LLM inference engine. The release adds support for `inp embd` in the server, enabling users to pass precomputed input embeddings directly into the token-generation process. This is a significant step for developers who want to integrate custom embedding layers or use embeddings from external models without first converting them to token IDs. The server now also accepts `embd` for sampled tokens, giving finer control over the sampling pipeline.
Additionally, this build fixes a bug in `~server_batch()` that could cause memory cleanup issues. The release ships with prebuilt binaries for a wide range of platforms—including macOS (Apple Silicon and Intel), Windows (CUDA 12/13, Vulkan, OpenVINO, SYCL), Linux (CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL), and Android arm64—making it easy to adopt on virtually any hardware. With over 122,000 stars and 21,200 forks on GitHub, llama.cpp continues to be one of the most actively maintained open-source LLM runtimes, and this update further cements its role for developers building local AI tools.
- New server support for `inp embd` lets you feed raw embeddings to generate the next token
- Adds `embd` handling for sampled tokens and fixes a `~server_batch()` cleanup bug
- Ships prebuilt binaries for macOS, Windows, Linux, and Android across CPU, CUDA, Vulkan, ROCm, OpenVINO, and SYCL
Why It Matters
Unlocks custom embedding pipelines in local LLM servers, expanding llama.cpp's usefulness for RAG and fine-tuned models.