llama.cpp b9767 boosts WebGPU MTP inference with mat-vec path for small batches

The latest release of llama.cpp, version b9767, brings targeted optimizations for WebGPU-based inference, particularly for multi-token prediction (MTP) workloads. The key change introduces a mat-vec (matrix-vector) multiplication path that is specifically tuned for small batch sizes, which are common in interactive AI applications like chatbots. By switching to this path, the GPU kernel avoids the overhead of full matrix-matrix operations, leading to significant speedups for latency-sensitive tasks. Additionally, a barrier was inserted into the NUM_COLS loop of the existing mul-mat-vec kernel to improve synchronisation and correctness when processing variable-length batches. These two changes together enhance the efficiency of running large language models directly in the browser or on consumer GPU hardware via WebGPU.

The release continues llama.cpp's tradition of broad platform support, with pre-built binaries available for Apple Silicon (macOS and iOS), Intel Macs, Linux (x64, arm64, s390x), Windows (x64, arm64), and Android arm64. GPU backends covered include Vulkan, ROCm, CUDA, OpenVINO, and SYCL, ensuring developers can deploy the optimizations across diverse hardware. For professionals using llama.cpp for local LLM inference, b9767 means faster response times when generating multiple tokens in parallel, especially on devices where small batch sizes are the norm. The mathematically lightweight mat-vec approach is a smart trade-off, leveraging the fact that many practical inference workloads involve sequences of a few tokens rather than large batches.