llama.cpp's new CUDA kernel speeds up NVIDIA DGX Spark inference
NVIDIA's commit tunes MMVQ with 8 warps for faster single-batch AI on GB10
NVIDIA's ynankani has landed a performance-focused commit (b10481) in llama.cpp, the popular open-source LLM inference engine with over 124k GitHub stars. The patch refines the CUDA implementation of MMVQ—the matrix-vector multiplication kernel used during autoregressive decoding—by setting nwarps=8 for batch size 1 (bs=1) on dense (non-MoE) models. This targets NVIDIA's DGX Spark, the compact desktop workstation powered by the GB10 Grace Blackwell chip. Because single-batch decoding is the common case for interactive chat and agent workloads, the new warp configuration better utilizes the GB10's GPU cores, cutting token-generation latency.
The commit also introduces a clever optimization for mixture-of-experts (MoE) models: it skips inactive expert computations based on k-geometry, permitting only a small "idle tail" of work that doesn't block subsequent layers. This reduces wasted CUDA kernel launches while preserving correctness. The code was renamed from "DGX Spark" to "GB10" parameters and fixed for MSVC constexpr lambda captures, ensuring cross-platform compilation. For developers running local LLMs on NVIDIA hardware, this means snappier responses on the DGX Spark and potentially other GB10-based devices. The change is a targeted micro-optimization—typical of llama.cpp's iterative approach—but it demonstrates how hardware-specific tuning can yield meaningful speedups in real-world inference.
- b10481 adds MMVQ with nwarps=8 for bs=1 on dense models, tuned for NVIDIA DGX Spark (GB10)
- Skips MoE experts using k-geometry to reduce idle tail latency in mixed-expert models
- Commit by NVIDIA's ynankani, part of llama.cpp's 124k-star repo, focused on faster single-stream decoding
Why It Matters
Hardware-specific CUDA tuning means faster, more efficient local LLM inference on NVIDIA's GB10 devices for professionals.