Knowledge Packs: Zero-Token Knowledge Delivery via KV Cache Injection
New research bypasses RAG's token waste by injecting pre-computed KV caches for zero-cost knowledge.
A new research paper by Andrey Pustovit introduces 'Knowledge Packs,' a novel technique that challenges the token inefficiency of standard Retrieval-Augmented Generation (RAG). The core innovation is the injection of pre-computed Key-Value (KV) caches directly into a transformer model's context. The paper proves a mathematical equivalence: for causal transformers, the KV cache generated from a forward pass on a factual text (F) is identical to the cache a joint pass on that text plus a question (F+q) would produce, due to the causal attention mask. This allows the same knowledge to be delivered at a 'zero token' cost, with experiments showing up to 95% token savings compared to prepending the full text.
Crucially, the research identifies that this equivalence is exact but fragile, with incorrect chat template formatting causing a 6-7 percentage point performance degradation. When applied correctly, the method produced zero divergences in outputs across 700 test questions on both Qwen3-8B and Llama-3.1-8B models. Beyond mere efficiency, the KV cache interface unlocks a second capability: behavioral steering. Because the Rotary Position Embedding (RoPE) rotates keys but leaves values untouched, researchers can apply contrastive 'deltas' to cached values to nudge model behavior subtly, while similar manipulations to keys destroy coherence. This steering effect is localized to mid-layer values (layers 33-66%), uses nearly orthogonal directions, and can be composed. Both knowledge delivery and behavioral steering channels can operate simultaneously without interference at an alpha blending parameter <= 0.7, all without any model training or weight modifications.
- Delivers knowledge at zero token cost by injecting pre-computed KV caches, saving up to 95% of tokens versus standard RAG.
- Achieved zero output divergence across 700 questions on Qwen3-8B and Llama-3.1-8B when chat templates are formatted correctly.
- Enables a second channel for behavioral steering by manipulating cached values in mid-layers (33-66%), composable and without model training.
Why It Matters
This could drastically reduce inference costs for knowledge-intensive AI applications and open new avenues for controlling model behavior post-training.