b8145
Latest llama.cpp release shows 24B parameter MoE model running efficiently on standard CPU hardware.
The llama.cpp team has released commit b8145, a significant update that adds official benchmarking support for the LFM2-24B-A2B model. This 23.84 billion parameter mixture-of-experts (MoE) architecture achieves 30.35 tokens per second on CPU with 10 threads when quantized to Q4_0 precision, consuming just 12.54 GB of memory. The benchmark compares directly against the smaller LFM2-8B-A1B model, which achieves 49.24 tokens/sec with 8.34B parameters.
Technically, this release demonstrates llama.cpp's continued optimization for efficient CPU inference of complex model architectures. The LFM2-24B-A2B represents a new class of MoE models that activate only subsets of parameters per token, offering better performance-per-parameter than dense models. The Q4_0 quantization reduces memory requirements by 75% compared to FP16 precision while maintaining reasonable accuracy.
This update matters because it brings advanced MoE architectures within reach of standard hardware. Developers can now run 24B parameter models on consumer CPUs without expensive GPU setups. The performance numbers suggest practical usability for local AI applications, from coding assistants to research prototypes. As MoE architectures become more prevalent in frontier models, tools like llama.cpp ensure they remain accessible beyond cloud-only deployments.
- Llama.cpp b8145 adds official support for LFM2-24B-A2B MoE model benchmarking
- 23.84B parameter model achieves 30.35 tokens/sec on CPU with Q4_0 quantization
- Enables efficient local inference of advanced architectures without GPU hardware
Why It Matters
Makes cutting-edge mixture-of-experts models practically usable on consumer hardware, democratizing access to advanced AI architectures.