Fibonacci sparse attention beats learned methods, extrapolates 4x without collapse

A new paper by Chad Capps (arXiv:2606.28560) proposes a sparse attention mechanism that uses Fibonacci-spaced offsets with a per-layer scalar alpha to compress or expand spacing. The author trained 21 language models under identical conditions (60M parameters, 512 hidden, 16 layers, 426M tokens) and compared four alpha schedules: fixed, per-layer learned, a static linear stagger, and a coprime reassignment—plus a reach-matched power-of-2 control. The results show that a static per-layer stagger consistently improves perplexity over both fixed and learned alpha, and the benefit is base-agnostic (applying the stagger to a power-of-2 base lifted it to parity with learned Fibonacci). Notably, learning alpha per layer added roughly 5x inference latency with no improvement over the static schedule.

The most impactful finding concerns extrapolation: all sparse variants maintained performance at 4x training sequence length with negligible degradation, while a dense baseline of the same recipe suffered a 201% perplexity increase. The author attributes this to sparse attention only attending to relative positions seen during training via fixed offsets. However, two honest negatives are reported: at training length, the best sparse model still has about 26% higher perplexity than the dense baseline, and the staggering gain is uniform across context positions rather than concentrated at long range. This suggests sparse attention is a strong candidate for scaling to very long contexts but at a baseline accuracy cost.