b8470
The latest update enables native BF16 support for CUDA, boosting speed on NVIDIA GPUs without precision loss.
The open-source project llama.cpp, maintained by the ggml-org team, has released a significant technical update with commit b8470. This commit introduces native BF16 (BrainFloat16) computation support for the critical Flash Attention operation within its CUDA backend. Previously, the software would convert BF16 data to FP16 before processing on NVIDIA GPUs, adding overhead and potentially losing precision. Now, the 'vec' and 'tile' kernels can process BF16 natively, which is the preferred precision for many modern AI models, leading to more efficient memory usage and faster matrix multiplications.
The update specifically enhances performance for users running large language models locally on NVIDIA GPUs that support BF16, such as the RTX 30 and 40 series. While the 'mma' (matrix multiply-accumulate) kernel still requires conversion—noted as a 'todo'—the change to the preceding attention kernels is a major step. The commit also includes fixes for CI failures on older Turing architecture GPUs and HIP (AMD's platform) compatibility, ensuring broader stability. For developers and enthusiasts, this means the popular tool for offline, CPU/GPU hybrid inference just got faster and more accurate for the latest model formats.
- Adds native BF16 computation for Flash Attention's vec and tile kernels in the CUDA backend, removing a conversion step.
- Improves inference speed and precision for models quantized to or using BF16 on compatible NVIDIA GPUs (Ampere, Ada Lovelace).
- Includes stability fixes for Turing GPUs and HIP platforms, broadening hardware support for the 98.9k-star project.
Why It Matters
Faster, more precise local AI inference makes running advanced models like Llama 3 more accessible and efficient for developers and researchers.