KIV: 1M token context window on a RTX 4070 (12GB VRAM), no retraining, drop-in HuggingFace cache replacement - Works with any model that uses DynamicCache [P]
New open-source system runs 1M token models on consumer GPUs without any model retraining required.
A new open-source project called KIV (K-Indexed V Materialization) is making waves by enabling massive context windows on consumer-grade hardware. Developed by Babyhamsta, KIV acts as a middleware layer that replaces the standard key-value (KV) cache in HuggingFace transformers with an intelligent, tiered retrieval system. The core innovation lies in separating the smooth, structured K vectors (used as search indices) from the chaotic V vectors (stored in system RAM). During each decode step, KIV uses the K vectors to retrieve only the ~256 most relevant V entries from RAM, dramatically reducing GPU memory pressure. This approach requires no model weight modifications, retraining, or distillation—it simply hooks into the HuggingFace cache interface as a drop-in replacement.
Test results are impressive: on an RTX 4070 with 12GB VRAM running Gemma 4 E2B (4-bit quantized), KIV achieves a 1 million token context window with just 12MB of VRAM overhead and ~6.5GB total GPU usage. Generation speed reaches 4.1 tokens/second at full 1M context, while maintaining 70/70 accuracy on needle-in-haystack tests up to 32K tokens. The system works with any model using DynamicCache architecture, including Gemma 4, Qwen2.5, TinyLlama, and Phi-3.5 across multiple attention variants (MQA/GQA/MHA). Current limitations include linear CPU RAM scaling (5.8GB at 1M tokens) and decode speed bottlenecks from CPU-to-GPU transfers, but the architecture shows significant promise for democratizing large-context AI inference.
- Enables 1M token contexts on RTX 4070 (12GB VRAM) with only 12MB VRAM overhead for cache management
- Achieves 4.1 tokens/second generation speed at full 1M context with 70/70 needle-in-haystack accuracy up to 32K tokens
- Drop-in HuggingFace replacement requiring no model retraining, tested on Gemma 4, Qwen2.5, TinyLlama, and Phi-3.5 models
Why It Matters
Democratizes large-context AI by making million-token models runnable on consumer hardware, potentially lowering inference costs and expanding accessibility.