RotaryQuant compresses 120B MoE models to run on 16GB RAM
New compression system fits massive 120B MoE models into consumer GPUs with near-zero accuracy loss...
Researchers from RotaryQuant (Anthony Lui, Mohamed Elsaied, and Neel P. Savani) have developed RotaryQuant, a breakthrough compression system that enables running massive 120-billion-parameter mixture-of-experts (MoE) language models on consumer-grade hardware. Published as arXiv:2608.08081, the system addresses three key memory bottlenecks: resident weight matrices, growing KV cache states, and paged expert sublayers.
The system combines three innovations: mixed-precision weight quantization (4-bit/2-bit/8-bit depending on layer), LRU-based expert offloading to disk, and IsoQuant - a novel KV cache compression method using Walsh-Hadamard transforms and SO(4) rotations before 3-bit quantization. A fused four-kernel Metal GPU pipeline processes attention directly on packed 3-bit tensors, eliminating full-precision KV state materialization. This enables running models like Gemma 4-26B, Qwen3-30B, and Nemotron-H 120B within 16-32GB RAM at 9-19 tokens/second with near-zero accuracy loss (≤0.0012 perplexity degradation) and full 32K context retention.
- RotaryQuant fits 120B MoE models like Nemotron-H into just 32GB RAM (16GB for smaller models) using three-axis compression
- IsoQuant reduces KV cache to 3-bit precision through Walsh-Hadamard transforms + SO(4) rotations, requiring only 256 parameters vs 16,384 for dense methods
- Achieves 9-19 tokens/sec generation with ≤0.0012 perplexity degradation and 100% retrieval accuracy at 32K context
Why It Matters
Bridges the gap between cutting-edge AI and accessible hardware, democratizing large model deployment for researchers and developers.