BaseRT on Apple M5 crushes LLM inference with 6.4x speed boost
New runtime exploits M5's Neural Accelerators for on-device AI that rivals cloud performance.
A new arXiv paper (arXiv:2607.19438) from researchers Fabian Waschkowski, Prabod Rathnayaka, and Lukas Wesemann presents BaseRT, a native Metal inference runtime that leverages Apple M5's redesigned GPU architecture—where every core packs a dedicated Neural Accelerator exposed via Metal 4's tensor API. The runtime adds hand-written tensor-core kernels (dense and MoE GEMM, flash-attention prefill) that route compute-bound matrix multiplications through these accelerators, while keeping memory-bound decode on existing specialized kernels.
On an Apple M5 Pro, BaseRT tested 15 model configurations from the Qwen3, Qwen3.5/3.6, Llama 3.2, and Gemma 4 families (sub-1B to 35B parameters). Results show up to 6.4× higher prompt-processing throughput than llama.cpp and 3.9× higher than MLX, with the largest gains on mixture-of-experts models. Decode latency also improved by up to 1.75× over llama.cpp and 1.33× over MLX. The work establishes a new performance ceiling for on-device LLM inference and is publicly available.
- BaseRT achieves up to 6.4× higher prompt throughput than llama.cpp and 3.9× over MLX on Apple M5 Pro.
- Hand-written Metal 4 tensor kernels exploit M5's per-core Neural Accelerators for compute-bound operations.
- Tested across 15 model configurations from Qwen3, Llama 3.2, and Gemma 4 families (sub-1B to 35B parameters).
Why It Matters
Apple Silicon now rivals cloud GPUs for local LLM inference, enabling faster, private on-device AI.