llama.cpp's b10355 adds multi-output sampling for faster token speculation

llama.cpp, the wildly popular open-source project that lets you run large language models locally on consumer hardware, just shipped release b10355. With over 123k stars and 21.5k forks, it's the go-to tool for local AI inference, and this release focuses on making speculative decoding faster and more consistent. The headline feature is multi-output backend sampling, which now allows the sampling process to happen directly on the backend (GPU, CPU, etc.) and supports token speculation. That means the model can speculate multiple tokens at once while sampling in a single pass, a significant speedup for interactive and streaming use cases. To support this, the maintainers added a numeric context parameter that declares the maximum number of outputs one sequence can produce, giving developers explicit control over batch sampling limits.

The release also includes a batch of reliability fixes. The mask sum is now clamped before being converted into the sampled index, preventing edge-case out-of-bounds issues during sampling. Memory reuse for output views has been disabled to avoid stale data across speculative batches. More importantly, the team worked hard to match the sampling distribution between CPU and GPU backends, fixing mismatches that could cause subtle differences in generation results depending on your hardware. They simplified the code, fixed tests on Vulkan, and rebased the changes cleanly. Binaries are available for macOS (Apple Silicon, Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Windows (CPU, CUDA, Vulkan, HIP), Android, and iOS, making the update easy to grab for nearly any setup. For local AI enthusiasts and developers, b10355 is a solid step toward faster, more consistent speculative decoding across all platforms.