Gaussian Mixture Attention slashes Transformer complexity to linear time

Standard dot-product attention scales quadratically with sequence length, a major bottleneck for long-context Transformers. Gaussian Mixture Attention (GMA) tackles this by replacing explicit pairwise query-key comparison with a probabilistic routing mechanism. Queries and keys map to posterior responsibility vectors over K learned Gaussian mixture components. Their overlap defines an implicit affinity, while values are written to and read from a K-slot latent memory. By exploiting matrix associativity, GMA avoids materializing the N×N affinity matrix, using two responsibility matrices with O(NK) storage instead of O(N^2). The authors formulate bidirectional and causal variants, provide differentiable parameterization, and analyze gradient structure and local routing stability.

Empirically, GMA demonstrates the intended fixed-K linear memory scaling and performs competitively with attention-style baselines on long-context classification tasks. Causal GMA improves over tested linear/random-feature attention variants on WikiText-103 but remains behind optimized causal softmax attention and Mamba in current implementation. Analysis of learned responsibilities shows broad component usage and moderate alignment with token categories, suggesting GMA offers a probabilistic, interpretable alternative rather than a universal replacement. The paper, with 55 pages, provides thorough theoretical grounding and opens a new direction for efficient sequence mixing.