Qwen 3.5 35B runs 1M context on RTX 3090 with KVarN quantization

In a Reddit post that quickly gained traction, user manu69x demonstrated something many thought impossible: loading a near-1M-token context into a single consumer-grade RTX 3090 with 24GB VRAM while keeping retrieval quality intact. The model in question was Qwen 3.5 35B A3B—a 35B-parameter mixture-of-experts model that activates only 3B parameters per token—taking just 17GB of VRAM. The breakthrough came from KVarN (Variance-Normalized KV-Cache Quantization), a technique from Huawei that downscales both K and V cache tensors to 4-bit precision with better accuracy than standard quants.

The setup relied on a custom BeeLlama.cpp fork (v0.4.3 preview) developed by Reddit user Anbeeld, who also shared KLD benchmarks showing KVarN outperforming conventional low-bit quantization for key-value caches. What makes this report remarkable is not just that the model survived a 1M-token context without crashing, but that it actually maintained coherence: manu69x planted 7 'needles' in different parts of the text and successfully extracted them all—a standard test for long-context memory. By contrast, q4 quantizations reportedly failed the same test. This suggests KVarN may close the precision gap that previously forced developers to choose between context length and accuracy, opening the door to running massive-context models on affordable hardware rather than clusters of A100s.