Developer Tools

llama.cpp b10353 fixes silent CUDA/Metal roll bug for accurate tensor ops

A subtle stride-handling flaw caused wrong results on GPU backends...

Deep Dive

llama.cpp, the wildly popular C++ library for running LLMs locally (123k stars on GitHub), has shipped release b10353 with an important correctness fix. The release addresses a bug in the ROLL tensor operation where CUDA and Metal backends would silently produce incorrect results for non-contiguous source tensors. The ggml_roll function only asserted that nb[0] equals the type size, leaving permuted tensors technically valid inputs. However, the CUDA and Metal kernel implementations indexed strictly by element count (ne) and never read the stride (nb) values, causing mismatched output without any warning.

The fix adds a contiguity requirement to both CUDA and Metal backends, matching the same guard already used for GGML_OP_ROPE. This ensures the scheduler now falls back to the CPU implementation when a non-contiguous source is passed, which correctly handles strides. The release also includes a new permuted test_roll case to catch this class of bug in the future. This is a subtle but significant fix for production users—silent numerical errors in tensor operations are the worst kind, as they corrupt model outputs without crashing. Anyone running llama.cpp on NVIDIA GPUs via CUDA or Apple Silicon via Metal should update to b10353 to avoid unpredictable results when their models or applications use non-contiguous memory layouts.

Key Points
  • Fixes silent wrong results in ROLL op on CUDA and Metal when src tensor is non-contiguous (permuted)
  • Adds contiguity requirement to both backends, matching existing GGML_OP_ROPE guard, forcing CPU fallback
  • Includes new permuted test_roll case; released Aug 10 with version b10353

Why It Matters

Prevents silent numerical corruption in GPU-accelerated LLM inference, a must-have update for llama.cpp users.

📬 Get the top 10 AI stories daily