llama.cpp v9948 slashes CUDA memory usage for top-k and argsort

llama.cpp, the popular open-source C/C++ inference engine for LLMs (120K GitHub stars), released version b9948 with a critical memory optimization for CUDA operations. The commit, led by creator Georgi Gerganov, modifies the CUDA implementations of `ggml_top_k()` and `ggml_argsort()` to process data in smaller chunks. Previously, these functions allocated large temporary buffers proportional to the full tensor size, causing out-of-memory errors on mid-range GPUs. Now, the computation is split into manageable pieces, drastically reducing peak memory usage.

This optimization is particularly impactful for users running quantized models (e.g., 4-bit or 8-bit) on consumer GPUs with 8–24GB VRAM. By lowering the memory footprint of ranking and sorting operations during token generation, the update allows for longer sequences, larger batch sizes, or higher precision outputs within the same hardware constraints. The change is transparent — no user configuration needed — and applies to all llama.cpp backends that use CUDA, including direct CUDA, CUB, and Vulkan. As LLM inference increasingly moves to edge devices, such targeted memory savings help democratize access to powerful models.