llama.cpp b10089 brings full CUDA quantized type support for GET_ROWS
All quantized types now run directly on GPU for embedding lookups.
llama.cpp's latest release, b10089, completes CUDA GET_ROWS type coverage by adding support for k-quants, i-quants, and mxfp4 quantization formats. Previously, certain quantized types (like Q4_K_M storing token_embd as q6_K) forced the backend to fall back to the host CPU, copying the full embedding matrix on every token. The new kernels factor super-block dequantizers into shared device functions and reuse them in a dedicated k_get_rows_kq kernel, handling one (dst row, super-block) pair per thread block with optimized thread layouts (32 threads for q4_K, 64 for others).
For i-quants, the same 32-thread layout from convert.cu is applied, with a guard that row size (ne0) must be a multiple of QK_K—except for iq4_nl and mxfp4, which use 32-value sub-blocks that don't guarantee alignment. The release also fixes a regression where the constraint was incorrectly applied to f16/f32/bf16/i32 and legacy quants. Now all quantized GGML types take the direct device path, accelerating local LLM inference for Apple Silicon, Linux, Windows, and other platforms. The update was contributed by pwilkin and is signed with GitHub's verified signature.
- Adds GPU-side GET_ROWS support for k-quants (Q2_K through Q6_K) and nine i-quants, eliminating CPU fallback for embedding lookups.
- Introduces shared super-block dequantizer functions reused from convert.cu kernels, with thread layouts optimized per quantization type (32/64 threads).
- Fixes a regression where row-size constraints were incorrectly applied to non-quantized types, restoring unconditional support for f16/f32/bf16/i32.
Why It Matters
Faster local LLM inference by keeping all embedding lookups on GPU, reducing memory transfers for quantized models.