Researchers unveil OptR for 2-bit KV-cache quantization
OptR cuts INT2 KV-cache errors by optimizing attention output directly
Researchers from Seoul National University and the University of Southern California have introduced **OptR (Output-aware Rotation)**, a novel method for INT2 KV-cache quantization that directly targets the error propagation bottleneck in long-context LLM inference.
Existing rotation-based quantization techniques optimize cache statistics or proxy errors before full attention readout, failing to account for the impact of errors propagated through the attention mechanism and output projection (W_O). OptR addresses this mismatch by decomposing post-W_O attention-output errors into key- and value-induced components and learning per-head orthogonal corrections through the full INT2 quantization and attention path.
OptR further introduces an attention-equivalent key reparameterization to eliminate large channel-wise offsets without altering the softmax distribution. In evaluations across three models and five reasoning/coding benchmarks, OptR consistently improves performance over QuaRot and OSCAR while enhancing long-context retrieval capabilities. Crucially, it maintains compatibility with standard paged KV-cache formats with negligible inference overhead.
- OptR is a rotation-based method for INT2 KV-cache quantization that minimizes post-W_O attention-output errors
- Improves QuaRot and OSCAR across 3 models and 5 benchmarks while maintaining paged KV-cache compatibility
- Introduces attention-equivalent key reparameterization to reduce channel-wise offsets without softmax changes
Why It Matters
Could enable 2-bit KV-cache quantization for LLMs with minimal accuracy loss, cutting memory costs for long-context inference.