llama.cpp b9670 fixes NVFP4 edge cases and LoRA dequant order
Critical precision fix for LLM inference on edge hardware
The open-source llama.cpp project released version b9670, addressing two critical issues in LLM inference. First, it fixes and restricts NVFP4 edge cases in llama-graph, a component handling compute graphs for AMD GPUs. Second, it moves post-GEMM MUL required for dequant before LoRA and bias addition. This ensures that LoRA adapters receive fully dequantized values (as literature suggests LoRA happens after multiplication but before bias addition), and that bias add for ModelOPT occurs on fully dequantized values. The commit references multiple GitHub issues (#24331, #23484, #8332) indicating community-driven troubleshooting.
The release builds on a wide range of platforms: macOS (Apple Silicon and Intel), Linux (x64, ARM64, s390x with CPU/Vulkan/ROCm/OpenVINO/SYCL), Windows (CPU, ARM64, CUDA 12 & 13, Vulkan, SYCL, HIP), and Android ARM64. Some configurations like openEuler and macOS KleidiAI are disabled in this release. These changes improve numerical accuracy for users running quantized LLMs with custom adapters, particularly on AMD hardware. The fix is essential for developers deploying local LLMs with LoRA fine-tunes or needing precise bias operations.
- Fixes NVFP4 edge cases in llama-graph for AMD GPU inference
- Adjusts dequantization order: LoRA now applies after multiplication but before bias add
- Supports 20+ platform configurations including CUDA 12/13, ROCm 7.2, Vulkan, SYCL, and ARM64
Why It Matters
Ensures numerical correctness for local LLM inference with LoRA adapters on diverse hardware.