llama.cpp b9630 adds Cohere2MoE support for TinyAya models
New release integrates Cohere's mixture-of-experts architecture into llama.cpp...
Llama.cpp, the popular open-source C++ implementation for running large language models locally, has released version b9630. This iteration introduces a key feature: support for Cohere2MoE models within its vocabulary handling, specifically targeting the TinyAya model family. Cohere2MoE refers to Cohere's implementation of mixture-of-experts (MoE) architecture, which activates only a subset of model parameters per token, enabling faster inference and lower memory usage without sacrificing quality. TinyAya is a compact MoE model optimized for edge deployment.
Beyond the new model support, this release continues llama.cpp's tradition of broad cross-platform availability. Pre-built binaries are provided for macOS (Apple Silicon with optional KleidiAI optimizations, Intel x64, and iOS framework), Linux (multiple CPU architectures with Vulkan, ROCm 7.2, OpenVINO, SYCL FP32/FP16), Windows (x64 and arm64 with CUDA 12/13, Vulkan, SYCL, HIP), and Android arm64. This ensures that developers and enthusiasts can run MoE models like TinyAya on everything from high-end GPUs to mobile devices, further democratizing local AI inference.
- Adds Cohere2MoE vocabulary support for TinyAya models in llama.cpp
- Provides pre-built binaries for 15+ platform combinations including macOS, Linux, Windows, and Android
- Enables running mixture-of-experts architectures locally on diverse hardware from GPUs to mobile
Why It Matters
Brings cutting-edge MoE model inference to local hardware, expanding accessibility of efficient AI architectures.