Unsloth releases GGUF of Cohere's 30B A3B model for local inference
Efficient 30B parameter model with 3B active parameters now runs locally via llama.cpp.
Deep Dive
A GGUF for the new Cohere 30B A3B model has been shared. The poster hasn't tested it yet but thinks it's related to a llama.cpp pull request (#24260) submitted by /u/jacek2023.
Key Points
- Cohere's 30B total parameter MoE model with only 3B active parameters per token
- GGUF quantization by Unsloth enables local inference via llama.cpp with custom kernel support
- Pull request #24260 adds architecture-specific optimizations for the A3B model
Why It Matters
Brings enterprise-grade MoE performance to local hardware, democratizing advanced code and reasoning AI.