llama.cpp b10025 relaxes CUDA tensor constraints for faster quantized concat
New release improves handling of non-contiguous tensors in quantized models on CUDA.
The llamp.cpp project, a popular library for running large language models locally, has released version b10025. This minor release focuses on optimizing quantized tensor operations by relaxing contiguity requirements for the concat operation specifically on CUDA. Previously, quantized concat required tensors to be memory-contiguous, which could force unnecessary data rearrangement. The new commit (a3e5b96) removes that restriction, allowing the library to handle non-contiguous quantized tensors directly without copying or transposing.
This change is co-authored by Stanisław Szymczyk and includes accompanying test cases to verify the fix. The release is built for a wide range of platforms: macOS (Apple Silicon, Intel), Linux (x64/arm64, with Vulkan, ROCm, OpenVINO, SYCL support), Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), Android (arm64), and openEuler. For AI practitioners using llama.cpp to run quantized models (e.g., 4-bit or 8-bit GGUF), this update reduces GPU memory overhead and improves throughput when models perform concatenation operations common in attention layers or model merging.
- Relaxes contiguity constraints for quantized concat on CUDA GPUs (CUDA 12 & 13).
- Co-authored by Stanisław Szymczyk with added test cases for non-contiguous tensors.
- Supports multiple backends: CUDA, Vulkan, ROCm, OpenVINO, SYCL, plus CPU and ARM.
Why It Matters
Optimizes local LLM inference on GPUs by reducing unnecessary memory copies, speeding up quantized model operations.