Why Attend to Everything? Focus is the Key

A research team including Hengshuai Yao, Xing Chen, and eight others has introduced Focus, a breakthrough method for making large language models dramatically more efficient without modifying their core weights. Unlike traditional attention mechanisms that approximate all token interactions, Focus learns which token pairs actually matter through learnable centroids that assign tokens to groups. This approach restricts distant attention to same-group pairs while maintaining full-resolution local attention, requiring training of only 148K additional parameters regardless of model size. The method works as a pure retrofit—all existing model weights stay frozen—yet improves domain perplexity with zero degradation on downstream benchmarks across five different attention architectures and model scales from 124M to 70B parameters.

At inference time, Focus delivers substantial speedups by restricting each token to its top-k highest-scoring groups, creating a hard sparsity pattern that yields 2x speedup while actually beating pretrained baselines on perplexity (41.3 vs 42.8 PPL). Remarkably, this pattern can be decomposed into two standard FlashAttention calls, achieving 8.6x wall-clock speedup at 1M tokens without requiring custom kernels. Unlike parameter-efficient fine-tuning methods like LoRA, Focus preserves model alignment—instruction-tuned models retain their TruthfulQA scores after adaptation, while LoRA degrades performance at every learning rate and rank tested. The method's Sinkhorn normalization enforces balanced groups as a hard constraint, and the resulting groups discover interpretable linguistic categories without any supervision.