New arXiv survey breaks down RoPE, ALiBi, and long-context scaling

Position encoding is the hidden backbone of every transformer—without it, self-attention can't tell token order. A new technical survey by Jiguo Li, published on arXiv (2608.10021), systematically unpacks the evolution of these methods, from sinusoidal and learned absolute embeddings to Shaw-style relative representations, Transformer-XL, T5 bias, ALiBi, and Rotary Position Embeddings (RoPE). The paper derives how RoPE transforms absolute positions into relative phase differences in Query-Key inner products, giving engineers a clear mathematical foundation for choosing encoding schemes. It evaluates each approach across four criteria: injection location, computational cost, KV-cache compatibility, and length extrapolation.

The survey then dives into long-context extensions that have become essential for modern LLMs—Position Interpolation, RoPE scaling laws, NTK-aware and Dynamic NTK scaling, YaRN, LongRoPE, and LongRoPE2. It examines frequency allocation, attention rescaling, training length, and target context length, summarizing practical implementation considerations while also covering evaluation protocols and position-encoding choices in representative LLMs. The central warning: being able to compute positional features beyond training length does not equal reliable generalization. Context extension must be validated with short-context retention, position-wise perplexity, retrieval, reasoning, and long-context code tasks. For practitioners, this is a structured reference that bridges theory and deployment decisions.