Tencent's HY3 295B MoE runs 2x faster than DeepSeek on Mac M5 Max 128GB
Open-source model achieves 32 tokens/sec decode on consumer hardware
Tencent's HY3, a 295B-A21B Mixture-of-Experts model, is making waves in the open-weight AI community by rivaling DeepSeek V4 Flash on benchmarks while running on consumer hardware. A user successfully deployed a 107GB dynamic 3-bit quant (UD128 style) on a MacBook M5 Max with 128GB unified memory, leveraging a custom llama.cpp build from PR #25395. After a minor GGUF architecture name fix ("hy-v3"→"hy_v3"), the model loaded in ~30 seconds and delivered impressive performance: 528 tokens/sec prefill and 32.4 tokens/sec decode on empty context, dropping to 124 and 16.3 at 16K context respectively. This represents roughly double the token generation speed of DeepSeek V4 Flash quantized similarly, with no perceived quality loss.
The achievement highlights the rapid maturation of open-weight large language models optimized for consumer GPUs. The inclusion of built-in speculative decoding (MTP) support in the PR promises further speedups once enabled. For developers and researchers running local inference, HY3 offers frontier-level reasoning at a fraction of the hardware cost, requiring only 128GB of unified memory (with a 122GB GPU memory ceiling) rather than expensive data center GPUs. The model's competitive benchmarks against DeepSeek V4 Flash make it a strong candidate for on-device AI assistants, code generation, and long-context analysis tasks.
- Tencent's HY3 295B-A21B MoE outperforms DeepSeek V4 Flash on benchmarks while running at 2x tokens/sec on a Mac M5 Max 128GB
- Achieves 32.4 t/s decode (empty context) and 16.3 t/s at 16K context using a 107GB IQ3_XXS quant via llama.cpp PR #25395
- Built-in speculative decoding module (MTP) is supported but hasn't been enabled yet, promising further speed gains
Why It Matters
High-quality open-source LLMs now run faster than proprietary models on consumer hardware, democratizing AI inference.