HGA enables 64K context on a single RTX 5090 with no retraining

Researchers have introduced Hierarchical Global Attention (HGA), a drop-in replacement for dense causal attention in pretrained long-context transformers. HGA preserves all original checkpoint parameters (W_Q, W_K, W_V, W_O) and requires no calibration or retraining. It uses a hierarchical two-level routing mechanism: first retrieving relevant chunks via compact RoPE-aware summaries, then refining selection by routing only the most relevant groups before performing exact token-level attention. This approach stores the full historical token K/V in host RAM or NVMe storage, transferring only a small routed working set to GPU memory. As a result, GPU memory consumption depends primarily on model weights and the working set, not on total context length.

Demonstrated on Qwen3-30B-A3B-Instruct-2507-FP8 running on a single RTX 5090 (32GB), HGA achieves a 64K-token context where token-level K/V storage would be infeasible. Across all tested context lengths (4K–64K), routed attention remains within approximately 0.01–0.02 nats of dense attention while the sparsity is only about 3%. The authors suggest that the approximation error is minimal and that remaining quality gaps are likely dominated by long-context positional encoding rather than the routing algorithm itself.