The Surprising Bottleneck in Running GLM-5.2 on 4×3090s: It's Not What You Expect

Zhipu AI's GLM-5.2, a 744B-parameter MoE model with 40B active parameters, has been successfully run on modest consumer hardware: 4× RTX 3090s (96GB total) paired with 192GB of DDR5-5600 RAM. Using unsloth's UD-IQ2_M quantization (2-bit) and llama.cpp's glm-dsa architecture (MLA + DeepSeek sparse attention), the setup achieves a decode speed of ~7.3 tok/s. The model uses explicit expert offloading: 19 of 75 MoE layers' experts reside on GPU (~83GB), while 56 layers are computed on CPU (~166GB resident).

Two A/B comparisons reveal critical insights. First, halving the quantization from IQ2_M to IQ1_M produced virtually identical speed (7.30 vs 7.29 tok/s) because the matmul shapes remain unchanged—CPU compute, not bandwidth, is the bottleneck. Second, increasing CPU threads from 6 to 12 yielded a 22% speedup (5.83 to 7.14 tok/s). Prefill remained flat at 135 tok/s regardless of threads. For users, the biggest real-world lever is turning off reasoning mode: a non-thinking inference completed in 13.5s versus 60-80s with high reasoning effort—a 5–6× wall-clock improvement. The x1 PCIe card proved useless for split inference but excellent as a sidecar for a separate model like Qwen3.6-35B.