b8776

The open-source project llama.cpp, maintained by ggml-org, has released a significant performance optimization in commit b8776. This technical update addresses a limitation in NVIDIA's CUDA programming model where the DeviceSegmentedSort algorithm cannot be captured within a CUDA Graph—a feature used to reduce launch overhead and improve performance. The commit forces the system to use the slower but graph-compatible DeviceSegmentedRadixSort only when necessary, allowing the faster DeviceSegmentedSort to be used in standard 'immediate mode' execution.

Benchmark results posted with the commit reveal tangible gains for AI workloads. On an RTX Pro 6000 Blackwell GPU, sorting operations (ARGSORT) on various tensor sizes saw performance improvements. For a 4096x512 element array, speed increased from 115.08 microseconds to 81.38 microseconds per run—a roughly 40% boost—while memory bandwidth utilization jumped from 135.77 GB/s to 192.00 GB/s. These sorting operations are critical during the inference process of large language models (LLMs) like Llama 3, affecting how efficiently the model processes attention and other tensor operations.

The update also includes a new test case to ensure the dispatch logic works correctly and is part of llama.cpp's extensive cross-platform CI/CD pipeline, which builds for macOS, Linux, Windows, and openEuler across CPU, CUDA, Vulkan, ROCm, and other backends. For developers and researchers running models locally, this low-level optimization translates to faster token generation, reduced latency, and better hardware utilization without any change to their code, simply by updating their llama.cpp library.