llama.cpp fixes GPU crash on low-shared-memory devices
GPUs with under 48 KiB shared memory no longer crash on MMQ prefill.
The latest llama.cpp commit (b10181) addresses a critical crash affecting GPUs with limited shared memory. The issue occurred in the MMQ (matrix multiplication quantization) kernel when per-block shared memory was below 48 KiB, the minimum for which MMQ tile configurations are designed. Devices like the Moore Threads MTT S70 (28 KiB) hit a GGML_ABORT during prefill (batch > 1) because no supported tile size fit. The fix disables MMQ on such devices by checking smpbo < 48 KiB, falling back to the BLAS path instead of aborting. After the patch, prefill throughput on the MTT S70 recovered to ~1470 t/s, matching a forced-CUBLAS build.
This is not MUSA-specific — any GPU with less than 48 KiB per-block shared memory benefits. The guard ensures stability without sacrificing performance on supported hardware. Token generation (batch=1) was never affected. The fix maintains compatibility with existing Q8_0, Q4_K_M, and FP16 quantizations. Developers using older or low-end GPUs can now run llama.cpp reliably, expanding local LLM inference to a broader range of hardware.
- Fixes GGML_ABORT on devices with <48 KiB shared memory (e.g., Moore Threads MTT S70 with 28 KiB)
- Disables MMQ path, falling back to BLAS, recovering prefill speed to ~1470 t/s
- Only prefill (batch>1) was affected; token generation remains unchanged
Why It Matters
Enables stable local LLM inference on low-shared-memory GPUs, expanding hardware support for llama.cpp.