llama.cpp's b10032 turbocharges CUDA inference with Lightning Indexer
New CUDA kernel leverages Tensor Cores for 2x faster attention indexing on Turing+ GPUs
Deep Dive
llama.cpp b10032 was released, adding a CUDA implementation for GGML_OP_LIGHTNING_INDEXER with generic vector and wmma kernels, as well as alignment checks for Q and K tensors. The release also includes builds for multiple platforms: macOS, Linux, Android, Windows, and others.
Key Points
- Adds CUDA GGML_OP_LIGHTNING_INDEXER with generic vector kernel and WMMA kernel for Tensor Cores
- Requires NVIDIA GPU with Turing architecture (RTX 20xx) or newer for WMMA optimization
- Includes alignment checks for Q and K tensors; supports builds on macOS, Linux, Windows, and Android
Why It Matters
Speeds up attention indexing on consumer GPUs, making local LLM inference faster and more efficient.