llama.cpp's -nkvo flag trades slight speed for massive context gains

The llama.cpp inference engine's -nkvo flag (short for --no-kv-offload) instructs the system to store the KV cache in system RAM instead of GPU VRAM. Many users avoid this option due to the expected performance hit, but one Reddit user running Qwen3.6 27B (IQ4_XS quant) on an RTX 5060 Ti 16GB and 32GB DDR5 found the trade-off surprisingly beneficial. Without -nkvo, fitting a 65k context required quantizing the cache to q4_0 and keeping only 58 layers on GPU, yielding 23 tps peak (16 tps during long generation). By enabling -nkvo, the entire model (99 layers) fits on the GPU with default f16 KV cache, dropping speed to only 19 tps peak (14 tps long) — a mere 4-5 tps penalty. Even more impressive, the same flag allows doubling the context to 131k tokens by keeping 63 of 65 layers on GPU, with nearly identical speed. Quantizing the RAM-offloaded cache offered no benefit, so users get f16 quality essentially for free.

These results challenge the conventional wisdom that offloading KV cache to RAM is always detrimental. For professionals running large models on consumer hardware with limited VRAM, the -nkvo approach provides a viable path to very long context windows (128k+ tokens) with only a minor throughput reduction — less than 20% in this case. The key takeaway: if you're compressing KV cache quality or reducing layers to fit a desired context length, try -nkvo first. The speed cost may be lower than the quality cost of aggressive quantization, especially when running on modern CPUs with fast DDR5 RAM and PCIe bandwidth.