UTokyo paper: Attention head spectra are fingerprints, not blueprints

A new paper from the University of Tokyo's Institute for Solid State Physics, authored by Li Hengyu, investigates the spectral properties of the attention operator M = W_q^T W_k. Because M is non-Hermitian, its eigenvalues are complex and its eigenvectors non-orthogonal. The study asks what these spectra encode, examining three levels—static, dynamic, and causal—across seven pretrained models using three positional schemes: RoPE, learned-absolute, and ALiBi.

Key results: Statically, previous-token heads under RoPE exhibit a rotational eigenvalue spectrum, while learned-absolute and ALiBi heads are non-rotational (content-like). This separation is perfect (p=0.029) and zeroing RoPE's phase channel eliminates induction in all three RoPE models. Dynamically, over Pythia training checkpoints, every head originates at the random-matrix null (Ginibre ensemble); the rotational signature appears after behavior, not before. Causally, no spectral channel is necessary: constrained two-layer training reroutes around every band, though with significant formation delay (q_BH <= 0.016). The cost structure reveals that imposing symmetry on learned-absolute models slows training by factor 2.9, while a RoPE head with fully symmetric static M still routes directionally via the phase channel—impossible under absolute positions. Conclusion: the positional scheme sets a default spectral algebra, a fingerprint sculpted after function, not a hard blueprint.