llama.cpp adds Tencent Hy3: 299B MoE with multi-token speculative decoding
New pull request integrates a 299B MoE model with MTP head for faster inference.
Deep Dive
A recent pull request for llama.cpp adds support for Tencent's Hy3 (HYV3ForCausalLM), a 299-billion parameter mixture-of-experts model with 80 layers and one multi-token-prediction (MTP) layer. It includes the model's MTP head as a draft-mtp speculative target for efficient decoding.
Key Points
- 299B parameter MoE model with 80 layers and 1 MTP layer.
- Multi-token-prediction (MTP) head used as draft model for speculative decoding.
- Integration enables faster text generation on consumer hardware via llama.cpp.
Why It Matters
Brings cutting-edge speculative decoding for massive MoE models to local inference, boosting speed.