Semidirect Fourier Delta Attention (SFDA) boosts long-context memory with phase control
New attention mechanism uses Fourier phases to solve recurrent state tracking limitations in linear attention.
A new paper on arXiv (cs.LG) from researcher Tiantian Zhang proposes Semidirect Fourier Delta Attention (SFDA), a novel linear attention mechanism that addresses a critical weakness in recurrent-state models: the inability to track exact state sequences and maintain long-context memory. SFDA is a phase-controlled generalization of the existing Kimi Delta Attention. Instead of using real diagonal decay (which compresses the KV cache into a fixed recurrent state), SFDA employs block-rotational Fourier control via complex exponentials: the state update equation introduces a diagonal matrix Λ_t with components α_t ⊙ e^{iθ_t}, enabling phase-dependent memory.
The paper's main theoretical contribution is a constructive chunk-WY factorization that allows products of the form A_t = Λ_t - u_t r_t* to be expressed as A_t ... A_1 = Γ_t - Y_t M_t W_t*, with rank growth bounded within fixed chunks. This yields an exact affine chunk transfer mechanism, formal stability and complexity bounds, and a compact characterization of phase-plus-low-rank memory. In toy state-tracking experiments, SFDA successfully learns cyclic memory patterns (e.g., counting modulo a number) while the phase-disabled baseline (KDA) remains near chance. The authors note that fused kernels and large-scale language model comparisons are left to future work, but the theoretical foundations open new avenues for designing linear attention that does not sacrifice exact state tracking.
- Replaces real diagonal decay with block-rotational Fourier control (phase parameters e^{iθ_t}) for richer memory dynamics.
- Constructive chunk-WY factorization ensures rank growth only within fixed chunks, enabling exact affine chunk transfer.
- In toy state-tracking experiments, SFDA learns cyclic memory (e.g., modulo counting) where the baseline KDA model fails, achieving near-perfect accuracy.
- Formal stability and complexity bounds are provided, suggesting O(1) memory per layer regardless of context length.
Why It Matters
SFDA could enable LLMs to handle extremely long contexts with exact cyclic state tracking, overcoming a key limitation of current linear attention.