Reddit community debates cheapest local setups for 70B-class LLMs like GLM 5.x

A recent thread on r/LocalLLaMA explores the most cost-effective ways to run large language models (LLMs) like GLM 5.x locally, specifically at 4-bit quantization (IQ4_XS). The original poster, running a Ryzen 5900X with 128GB DDR4 and a Radeon 7900XT (20GB VRAM), notes they can only handle models up to MiniMax 2.7B at Q4_K_S or Qwen 3.6 27B at IQ4_XS. They ask the community for alternatives that avoid expensive unified memory systems (e.g., Apple's M-series or NVIDIA's Grace Hopper).

Responses highlight several approaches: CPU-only inference using Sapphire Rapids Xeon processors with DDR5, which can handle moderate-sized quantized models through RAM; multi-GPU setups with partial or full offloading using tools like llama.cpp or oobabooga; and older server hardware (e.g., dual Xeon with high memory bandwidth). The thread reveals a growing interest in running 70B+ models without enterprise budgets, with users sharing real-world benchmarks for throughput and memory usage. Key trade-offs include inference speed vs. cost, with CPU-only setups offering lowest cost but slowest tokens per second, while multi-GPU offloading balances price and performance.