Dual Xeon CPU with 384GB RAM Runs GLM-5.2 at 4.5 Tok/s — But There's a Catch
A Reddit user ran the frontier GLM-5.2 model entirely on CPU with impressive but slow results.
Deep Dive
A Reddit user (u/_TheWolfOfWalmart_) runs the UD-Q2-K_XL quant locally on a Dell PowerEdge R740 with dual Xeon 6248R CPUs (24 cores each) and 768GB RAM, using a single NUMA node (384GB) and ik_llama.cpp. With MTP drafting, generation reaches 4–5.5 tok/s in basic chat, dropping to ~3 tok/s in coding tasks. The 1M context fits fully in RAM, offering frontier-level coding assistance despite limited speed.
Key Points
- GLM-5.2 runs on dual Xeon 6248R (24 cores each) with 384GB RAM allocated via a single NUMA node
- Generation speed: 4–5.5 tok/s in chat with MTP drafting, ~3 tok/s in coding tasks
- 1M context window fits entirely in RAM; ik_llama.cpp used for CPU-only inference improvements
Why It Matters
Demonstrates that frontier AI models can run locally on CPU, opening possibilities for privacy-sensitive and offline professional workflows.