llama.cpp b9591 optimizes MTP with padding removal and faster copies
New release slashes GPU overhead by eliminating redundant D2D copies for multi-token prediction.
The latest release of llama.cpp, tagged b9591, brings a significant performance optimization to its multi-token prediction (MTP) feature. The core change removes padding and multiple device-to-device (D2D) copies that previously occurred when handling the recurrent cache snapshots. Instead of inferring the snapshot count K from the state dimensions, developers refactored ggml_gated_delta_net to accept only the initial recurrent state (D, 1, n_seqs) and pass K as an explicit op parameter. All emitted snapshots are now copied into the recurrent cache with a single strided ggml_cpy, dramatically reducing GPU memory operations.
This update affects all backends, including CPU, CUDA, Vulkan, ROCm, OpenVINO, SYCL, and KleidiAI. The release includes extensive platform support: macOS (Apple Silicon & Intel, iOS XCFramework), Linux (x64, arm64, s390x with various GPU backends), Windows (x64 CPU, CUDA 12/13, Vulkan, HIP), and Android arm64. CI builds were also fixed for macOS/iOS and Windows. With 116k stars and 19.5k forks on GitHub, llama.cpp remains the leading open-source solution for running large language models locally, and this update continues to push the boundaries of efficiency for on-device AI inference.
- Removed padding and multiple D2D copies for MTP, simplifying memory transfer with a single strided ggml_cpy.
- ggml_gated_delta_net now takes only the initial recurrent state (D, 1, n_seqs) with snapshot count K as a parameter.
- Available across all major platforms: Linux, Windows, macOS, iOS, Android, and various GPU backends (CUDA, Vulkan, ROCm, etc.).
Why It Matters
llama.cpp’s MTP optimization reduces GPU memory bottlenecks, making local LLM inference faster and more accessible for developers and users.