Your Phone's AI Just Got Faster Thanks to a Clever Split
This tech tweak could make AI on your devices snappier and more power-efficient.
Deep Dive
A hexagon update (pull request #31055) reworks KV-head splitting for multicore HMX flash attention. It merges the FA_HEAD_SPLIT and FA_SPLIT options into a single GGML_HEXAGON_FA_SPLIT flag: set to 0, it's off and only token-split applies; set to 1, it uses head-split, plus team-split when n_kv_heads is less than N_cores — so stranded cores still participate, as in the cited example of gemma-4 with 2 KV heads on 4 cores. Where heads divide evenly across cores, plain head-split is used.
Key Points
- New 'hexagon' update makes AI run faster on multi-core phones by splitting work more evenly.
- It ensures all cores are used, even when the AI model has fewer attention heads than cores.
- This could lead to quicker AI responses and better battery life on your devices.
Why It Matters
Faster, more efficient AI on your phone means quicker apps and longer battery life.