New uncertainty-gated router boosts long-context AI recall by 28 points

Standard block-sparse attention scales long-context language models by replacing expensive softmax with a per-query top-k selection over key blocks. However, this cutoff is myopic: when the k-th and (k+1)-th blocks are nearly tied in score, the selector commits without spending extra budget, and a dropped block that carries answer evidence becomes unrecoverable downstream. Thomas Rossi proposes a value-of-information router that measures, for each query, how decisively the top-k cut was made. If the gap is small, the router doubles the kept set for that query. The rule is backbone-agnostic and stacks with existing block-scoring methods such as Quest.

Extensive evaluations show consistent gains. On LongBench-v2 medium (n=215), router-on-Quest achieves paired recall 0.75 versus 0.47 for standard top-k (+28pp, p<0.01). On RULER NIAH multikey, it lands within 2 percentage points of dense attention at the same context length. The lift reproduces across four models from three architectures (Qwen2.5, Mistral-Nemo, Qwen3.6). At 128K context, the router preserves 0.81 and 0.89 of dense accuracy on Qwen2.5-7B-1M and Qwen3.6, respectively, compared to 0.09 for standard top-k on the former. Crucially, the fused selection-plus-kernel pipeline runs at 0.62x to 0.80x of dense wall time, making the improvement practical for deployment.