Researchers improve Linear Attention with QED for longer context
New 'Query-derived Erase Direction' doubles usable context length in linear attention models
Researchers Dhruman Gupta, Aritra Das, and Debayan Gupta have proposed a novel approach to improve linear attention mechanisms in machine learning models. Their paper, titled 'The Query Knows What to Forget: A Second Erase Direction for Linear Attention' (arXiv:2608.13668), introduces the Query-derived Erase Direction (QED), which addresses a critical limitation in existing delta-rule models like Gated DeltaNet-2 (GDN-2).
Traditional linear attention models derive their erase vectors from the key of the current token, but interference in reads is measured through the query. This mismatch means the erase step cannot effectively target query-based interference. QED solves this by adding a second erase direction derived from the query, orthogonal to the key. In the fast-weight view, this allows the model to cancel old-state content measured along the query, significantly improving retrieval performance at every length past the training window. Empirical results show QED approximately doubles the usable context length on the S-NIAH-1 benchmark.
- QED introduces a second erase direction derived from the query, orthogonal to the key, in linear attention models
- Improves retrieval performance at every length past the training window and doubles usable context length on S-NIAH-1
- Addresses interference degradation in long-context retrieval by enabling more precise cancellation of old-state content
Why It Matters
Enables linear attention models to handle much longer contexts efficiently, unlocking new capabilities for long-form document processing and extended dialogue systems.