Tencent's Hy3 and NVIDIA's audio MoE get precise GGUF quants
New imatrix quants for two massive MoE models with full reproducibility.
LordNeel has released comprehensive GGUF quant sets for two cutting-edge MoE models. For Tencent's Hy3 (295B parameters, 21B active, 262K context), quants range from Q6_K (226 GiB, effectively lossless) to IQ2_M (90 GiB, fits single 96GB GPU). The Q4_K_M at 167 GiB is recommended as a default. All quants include imatrix calibration, KL-divergence and top-token agreement metrics measured against BF16 reference, and llama-bench throughput numbers. The quant sets are fully reproducible from the files in the repo.
Meanwhile, NVIDIA's Nemotron-Labs-Audex-30B-A3B (30B hybrid MoE, ~3B active, 1M context) brings audio understanding and generation capabilities. Two tracks are provided: text-only GGUFs for standard llama.cpp use, and audio quants requiring a sidecar with NV-Whisper encoder, causal speech decoder, and enhancement VAE. Text quants show up to 2x generation speed improvement over BF16 (e.g., Q4_K_M at 345 tok/s vs 177). A notable MXFP4_MOE quant reduces size to 16.7 GiB with experimental quality. The audio pipeline is not yet integrated into a single GGUF runtime. Both model sets are important for democratizing access to large MoE models through efficient quantization, with full benchmarks and transparency.
- Hy3's Q4_K_M quant (167 GiB) fits on 2x 96GB GPUs with 90.0% top-token agreement and 67.3 tok/s.
- NVIDIA's audio model has separate text-only quants (e.g., Q4_K_M at 22.8 GiB, 345 tok/s) and audio quants needing additional components.
- All quants use imatrix quantization and include KL-divergence vs BF16 reference for reproducible quality assessment.
Why It Matters
Efficient, quantized MoE models enable powerful AI on consumer hardware, bridging accessibility and performance.