Developer Tools

llama.cpp v9933 fixes OpenCL Q6_K bugs and memory corruption

Memory corruption fixed for non-standard model weight dimensions on Adreno GPUs.

Deep Dive

llama.cpp, the widely-used C/C++ inference engine for LLMs, has released version b9933 with two critical OpenCL bugfixes. The first addresses garbled output for Q6_K quantized models on Adreno GPUs when the weight tensor's first dimension (ne01) is not a multiple of 128. This manifested in models like granite-3.1-3b-A800M-instruct whose vocabulary size is odd, causing the decode path to use an incorrect GEMM/GEMV kernel. The fix routes those cases to a correct flat GEMV for single-row inference (ne1==1) and falls back to CPU for batch inference (ne1>1) since no verified small-batch OpenCL kernel exists for those shapes. All standard hidden/FFN/vocab dimensions (which are powers of two) continue to use the fast path.

The second fix tackles a subtle memory corruption bug in the OpenCL buffer allocator. When carving quantized weights into per-component subbuffers (e.g., d, q, ql, qh), rounding each subbuffer's origin to the device alignment could cause the last subbuffer to extend past the tensor's allocated bytes. This allowed a neighboring tensor's upload to silently overwrite the previous tensor's data, corrupting model weights. The bug affected any quant type with component sizes not aligned to the device's base address (typically 128 bytes). The fix implements a custom get_alloc_size function that reserves worst-case slack (up to 512 bytes per quantized tensor) for the five-component q5_K scheme. Standard power-of-two dimensions remain unaffected. Both fixes were contributed by Li He from Qualcomm.

Key Points
  • Fixes garbled Q6_K output on Adreno GPUs for weight dimensions not multiples of 128 (e.g., odd vocab sizes).
  • Resolves memory corruption caused by misaligned quantized subbuffers that could silently corrupt neighboring tensors.
  • Reservation of up to 512 bytes per quantized tensor in OpenCL allocation ensures safe subbuffer carving without performance impact on standard shapes.

Why It Matters

Ensures stable Llama.cpp inference on mobile/edge Adreno GPUs, especially for non-standard model architectures.

📬 Get the top 10 AI stories daily