b8261

The open-source project llama.cpp, maintained by ggml-org, has released a significant performance optimization in commit b8261. The update extends the optimized `mul_mv_ext` (matrix-vector multiplication extended) kernel to support the BF16 (BrainFloat16), Q2_K, and Q3_K quantization data types for small batch sizes (2-8). Previously, operations using these specific data types were forced to use a slower, single-row processing path, creating a performance bottleneck. Now, BF16 operations will use the same efficient `float4` dequantization path as FP16, while Q2_K and Q3_K will use the high-performance `float4x4` K-quant path shared by Q4_K, Q5_K, and Q6_K.

This technical improvement has a direct impact on inference speed for users running quantized large language models (LLMs) like Llama 3 or Mistral on consumer hardware. The commit is part of the continuous optimization of the library's Metal backend for Apple Silicon (arm64) Macs and iOS devices, where efficient small-batch processing is crucial for responsive applications. By closing this performance gap, developers and researchers can achieve faster token generation and lower latency when using these popular, memory-efficient quantization formats, making local AI more practical.