New CPU-GPU hybrid system brings cloud-grade MoE inference to local hardware

Researchers from Tsinghua University have published a paper (accepted at OSDI '26) detailing a CPU-GPU hybrid system that brings cloud-grade service-level objectives (SLOs) to local deployment of large Mixture-of-Experts (MoE) models. The system tackles four key gaps in local inference: reliance on reduced-capacity models, inability to meet 30-second time-to-first-token (TTFT) for long prefills (over 12K tokens), sub-baseline decode throughput (under 20 tokens/s), and poor concurrency under mixed workloads. It introduces stream-loading prefill (SLP) achieving 1,200 tokens/s on commodity CPUs, and distributed SLP (DSLP) with expert parallelism hitting 1,800 tokens/s on two RTX 5090s, enabling 32K–45K token prompts within 30 seconds. Additionally, an AVX-512-optimized FP8 GEMV kernel delivers 4-5x lower CPU latency, and fine-grained CPU parallelism attains 28 tokens/s on INT4 DeepSeek-V3 (21.5 tokens/s on intact FP8 V3).

The system also uses intra-node prefill-decode disaggregation with zero-copy shared weights and a dual-batch attention-MoE overlap scheme, sustaining concurrency with under 15% latency increase and 50% throughput gains. This design allows local hardware to match cloud-quality inference for flagship MoE models without datacenter infrastructure, using original-precision models rather than quantized or distilled versions. The work reshapes local deployment by offering cost-effective, high-quality access to large language models on consumer CPU-GPU platforms, potentially democratizing advanced AI inference for researchers and smaller organizations.