llama.cpp b9536 brings major OpenCL optimizations for GPU inference
OpenCL kernels get 2x speedups for key operations on AMD and Intel GPUs
Deep Dive
The llama.cpp project released b9536, focusing on OpenCL backend improvements. Changes include enhanced get_rows, cpy, concat operations, and a reworked flat q6_K gemv kernel. Specifically, added multi-workgroup support for large rows, improved small cpy, packed concat for small inputs, and tweaked flat q6_K gemv with increased N_DST and remapped threads.
Key Points
- OpenCL backend now supports multiple workgroups for large row operations in get_rows
- q6_K flat gemv kernel reworked with increased N_DST and thread remapping for better cache performance
- Small cpy and packed concat for small inputs optimized to reduce latency
Why It Matters
Makes LLM inference faster on AMD/Intel GPUs, expanding open-source AI access beyond NVIDIA.