Developer Tools

llama.cpp b10327 fixes CUDA kernel bug for quantized copies

This release patches a thread/block miscount in quantized cpy kernels.

Deep Dive

llama.cpp's latest release, b10327, addresses a subtle but important bug in the CUDA implementation of quantized copy (cpy) kernels. The issue, fixed in pull request #26731, involved incorrect thread and block counts when launching quantized copy operations, particularly when tensor dimensions didn't align neatly with the default block size. This could lead to silent data corruption or kernel failures in edge cases, impacting the reliability of local LLM inference on NVIDIA GPUs. The maintainers also added a regression test for uneven block counts, ensuring the fix holds across a wider variety of tensor shapes.

The new binaries are available for nearly every major platform: macOS (Apple Silicon via arm64, both standard and KleidiAI-enabled builds, plus Intel x64), Linux (x64, arm64, s390x, with Vulkan, ROCm 7.2, OpenVINO, and SYCL backends), Android (arm64 CPU), and Windows (x64, arm64, CUDA 12 and 13 prebuilt DLLs, Vulkan, OpenVINO, SYCL, and HIP). Some niche builds like macOS Intel KleidiAI and openEuler variants are disabled for now. For AI engineers and hobbyists running quantized models like Llama 3 or Mistral via llama.cpp, this patch improves stability when dealing with non-power-of-two tensor dimensions—a common scenario in real-world model architectures.

Key Points
  • Fixes thread/block count in CUDA quantized cpy kernel launches (#26731)
  • Adds regression test for uneven block count copy cases
  • Ships prebuilt binaries for Windows, Linux, macOS, Android, and more

Why It Matters

Local LLM inference on NVIDIA GPUs gets more reliable, especially for quantized models with odd tensor shapes.

📬 Get the top 10 AI stories daily