On the Invariants of Softmax Attention

In a new paper posted on arXiv, Wonsuk Lee tackles a fundamental question about transformer attention: what properties hold universally across all softmax attention mechanisms, regardless of model or input? The answer is surprisingly rich. Lee defines the 'energy field' — the row-centered version of the attention logit matrix — and shows it obeys two classes of invariants.

Mechanism-level invariants are baked into the algebra: each row of the energy field sums to zero, its rank is bounded by the head dimension (typically 64–128), and this forces a specific spectral signature. Model-level invariants are not mathematically required but appear in every autoregressive language model tested (spanning GPT, LLaMA, and Mistral families). The energy field spreads its variance across all key positions rather than concentrating on a few. Lee traces this 'delocalization' to a property of the key matrix he calls 'key incoherence.'

The paper is remarkable for its generality: all results hold at multiple context lengths and across diverse inputs. Practical implications are immediate. The rank bound implies attention logits live in a low-dimensional subspace, which could lead to compression or more efficient attention approximations. More directly, key incoherence acts as a per-head training monitor — sudden divergence from incoherence could signal training instability. The work is a rare combination of mathematical depth and empirical validation, providing a new lens for understanding how tens of billions of parameters produce coherent attention patterns.