GLM-5.2 on B200: NVFP4 + 2x TP=4 replicas doubles throughput vs FP8 TP=8

A detailed deployment analysis for GLM-5.2—a 750B parameter MoE model with 256 experts, top-8 routing, and ~40B active parameters—on 8x B200 SXM nodes reveals that the optimal configuration is not the obvious tensor-parallel (TP) = 8 across the whole node. The key insight: MoE decode is bandwidth-bound, not compute-bound. Using NVFP4 precision (which halves weight bytes to 459 GB over FP8's 744 GB) allows each half-node (4 GPUs) to hold the entire model with ample KV cache headroom (~230 GB per replica). This enables two independent TP=4 replicas behind a load balancer, delivering an estimated aggregate 33,000 tok/s versus ~15,600 tok/s with a single FP8 TP=8 engine—roughly a 2x node-level throughput improvement.

The published single-replica benchmarks (source: InferenceX / SemiAnalysis) show NVFP4 TP=4 achieving 4,116 tok/s per GPU at 128 concurrent users (56.7 ms TPOT), compared to FP8 TP=8's 1,947 tok/s per GPU at 256 users (84.2 ms TPOT). Caveat: these arithmetic projections assume no scheduler or NCCL contention from dual replicas; real-world tests are still pending. At SemiAnalysis's $1.95/GPU/hr TCO, the NVFP4 setup costs ~$0.13 per million tokens—a 3.5x perf/$ advantage over H200 FP8 ($1.06/M tokens). The analysis also notes that for 1M context workloads, disaggregated prefill is mandatory due to memory monopolization, and the IndexShare attention mechanism's claimed 2.9x FLOP reduction at long context lacks independent TTFT measurements.