GLM-5.2 NVFP4 quant runs on 4x DGX Sparks at 128K context with 15 tps
A 410GB model from 1.5TB runs on four GB10 nodes using NVFP4 quantization.
Mapika's GLM-5.2 NVFP4 quant leverages NVFP4 quantization for MoE expert FFNs (routed + shared) while keeping attention (MLA + DeepSeek-style DSA indexer), router, and LM head in BF16. This reduces the model from 1.5TB to 410GB. The setup uses four standard NVIDIA-brand GB10 DGX Sparks connected via a Microtik RoCE switch. A custom vLLM fork with DCP (decode-context parallelism) across 4 TP ranks, combined with B12X sparse MLA patches and MTP1, makes 128K context feasible. Achieved ~512 tokens/s prefill and ~14.5-15.2 output tok/s on short-prompt codegen, with decode falling to ~13 tps at long context.
Critically, the Spark cluster outperforms a Mac M3 Ultra 512GB running the unsloth Q4_K_S quant. The Mac lacks MLA kernel support, causing decode to collapse severely as context grows, whereas the DGX Spark setup maintains consistent throughput. This demonstrates that even niche, hacky multi-node configurations can serve real-world inference for massive MoE models. The win combines aggressive system/Ray memory trimming with DCP4 sharding. This approach opens the door for running 400B+ models on modestly priced hardware clusters without sacrificing accuracy.
- Checkpoint shrunk from 1.5TB to 410GB (3.7×) via NVFP4 for MoE layers and BF16 for attention/router/LM head, with GSM8K accuracy within ~2 points of BF16.
- 4x DGX Sparks (GB10) with vLLM fork, DCP4, MTP1, and fp8 KV cache achieve 128K context and ~15-16 tps decode, compared to Mac M3 Ultra which suffers from MLA kernel-induced decode collapse.
- Setup uses RoCE switch, explicit 1.81 GB/rank KV cache, and 512 tokens/s prefill; demonstrates viable serving for large MoE models on multi-node consumer-grade hardware.
Why It Matters
Proves that multi-node GB10 clusters can serve 400B+ quantized MoE models efficiently, challenging the need for expensive single-GPU solutions.