Blurry Window Attention achieves 8x state efficiency over Sliding Window
New attention mechanism uses Dirichlet kernels to reconstruct blurry KV history efficiently.
Transformer language models face a bottleneck in long-context scenarios due to the quadratic complexity of Softmax attention and growing KV cache size. Alternative architectures like State-Space Models (SSMs), Linear Attention, and Attention with Bounded-memory Control (ABC) offer linear complexity and finite state, but struggle on retrieval and recall tasks. In a new arXiv paper, Axel Laborieux and colleagues introduce Blurry Window Attention (BLA), an ABC method that bridges this gap. BLA stores a frequency window from which a blurry KV history is reconstructed via interpolation using Dirichlet kernels. This design generalizes Sliding Window Attention (SWA) based on kernel resolution and can be seen as a special case of Gated Slot Attention with Dirichlet-based decay.
BLA's efficient implementation leverages Dirichlet kernels to maintain a finite state size while approximating full attention with blurring. On the MQAR synthetic benchmark, BLA achieves 8x better state efficiency than SWA and is competitive with leading linear attention models. In the RegBench task, only BLA and SWA among linear models improve performance as state size grows. This suggests BLA retains more relevant context for recall-heavy tasks. The work offers a promising path for long-context LLMs without sacrificing retrieval accuracy.
- BLA uses Dirichlet kernels to interpolate a blurry KV history from a stored frequency window.
- On the MQAR task, BLA achieves 8x better state efficiency than Sliding Window Attention.
- BLA and SWA are the only tested linear models that improve with larger state size on RegBench.
Why It Matters
Efficient long-context processing that maintains retrieval performance—key for RAG, document analysis, and memory-intensive AI applications.