llama.cpp b10076 cuts GPU memory reads 27% with vectorized CUDA
New CUDA kernel vectorizes int4 copies, slashing gather latency by 5.6μs on AMD Strix Halo.
**llama.cpp b10076** brings a significant CUDA optimization to the get_rows operation, which is critical for attention layers in transformer models. The previous scalar kernel (k_get_rows_float) copied one element per thread and recomputed row-invariant work (index loads, fast_div_modulo, row pointers) for every element. The new release introduces a vectorized path (k_get_rows_float_vec) that copies 16 bytes per thread via int4, hoisting invariant computations outside the per-element loop.
The vectorized path is compile-time gated on same-type src/dst (no casting required) and runtime-checked for 16-byte alignment of base pointers, row strides, and ne00 divisibility by 4. To prevent performance regressions on small gathers where block count drops below device CU count, an occupancy gate automatically falls back to the scalar path. On AMD Strix Halo (gfx1151), the DeltaNet recurrent-state gather (ne00=524288) drops from 18.6μs to 13.0μs (rocprofv3 timestamps), even outperforming the Vulkan backend. The overall get_rows benchmark shows a 27% reduction. All 47 test-backend-ops GET_ROWS tests pass.
This release also bundles prebuilt binaries for macOS (Apple Silicon with optional KleidiAI), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), Android (ARM64 CPU), and more. The commit and release notes credit assistance from Claude Opus 4.8.
- New vectorized CUDA kernel copies 16 bytes per thread (int4) instead of 1 element per thread
- DeltaNet recurrent-state gather on Strix Halo drops 18.6μs → 13.0μs (30% faster), total get_rows -27%
- Fallback occupancy gate prevents regression on small gathers; all 47 test-backend-ops GET_ROWS tests pass
Why It Matters
Speed up large model inference on GPU by 27% for memory-bound gather operations, directly improving token generation latency.