Running GLM 5.2, Kimi 2.7, DeepSeek V4 Pro on 4x-8x RTX 6000 PRO GPUs

A Reddit user running 4 to 8 RTX 6000 PRO GPUs (total 384-768GB VRAM) is seeking community feedback on running the latest large language models: GLM 5.2, Kimi 2.7, and DeepSeek V4 Pro. They note that in theory, these models can be loaded at 4-bit quantization on their hardware, but 8-bit may be out of reach. The core question is whether 4-bit quantization significantly degrades performance for agentic or programming use cases, as some reports suggest a high performance hit vs 8-bit for smaller models. The user also asks about backend choices (vLLM vs SGLang) and references a benchmark repository that lacks data on these newest models.

This discussion highlights the growing tension between model size and hardware constraints. With models like GLM 5.2 and DeepSeek V4 Pro pushing past 300B parameters, even 8x RTX 6000 PROs (768GB) struggle with 8-bit precision. The community's insights could guide professionals building local inference rigs for coding agents and autonomous tasks, where latency and accuracy are critical. The post underscores the need for more systematic benchmarks on quantization tradeoffs for complex agentic workflows.