AI Chatbot Runs Faster on Macs Thanks to Code Cleanup
Your next AI chat could be quicker and smoother, even on a laptop.
Deep Dive
A change in llama.cpp's ggml metal backend removes the mul_mm_id amax rescaling (#30924). According to the commit, the src1 power-of-two rescale in kernel_mul_mm_id is superseded by the per expert-token amax scaling done in the compute graph (#30861), which keeps the MoE down-projection activations within f16 range before the op runs. The commit drops the two-stage reduction kernels, their scratch buffer and the GGML_PREC_F32 dispatch added in #26223, as well as the test cases that drive MUL_MAT_ID src1 out of f16 range.
Key Points
- The update simplifies how AI models handle certain math, reducing errors and speeding up processing.
- It affects 'mixture of experts' models, which are used in some of the most advanced AI chatbots.
- You'll likely see faster and more reliable AI responses on your personal computer.
Why It Matters
Faster, more reliable AI on your own devices means you can get answers quicker without relying on the cloud.