llama.cpp adds int8 dp4 optimization for Adreno GPUs, boosting MoE prefill
New OpenCL kernel speeds up AI inference on Qualcomm Adreno GPUs by up to 2x
Deep Dive
llama.cpp added int8 dp4 dense and MoE prefill optimization for Adreno GPUs in OpenCL (PR #25537), with code refactoring and co‑authoring from Qualcomm engineer Li He.
Key Points
- llama.cpp merged int8 dp4 dense and MoE prefill kernels for Adreno GPUs (PR #25537)
- Optimization uses Qualcomm's int8 dot-product instructions, critical for quantized AI inference
- Patch co-authored by Qualcomm engineer Li He, ensuring close hardware-software alignment
Why It Matters
Enables 2x+ faster local LLM inference on millions of Snapdragon Android devices using quantized models.