New AI Research Makes Language Models Smarter by Reading Smarter

A new paper introduces Asymmetric Attention Heads (AAH), a framework that treats context length as a per-head allocation variable instead of giving every attention head the same full context. The heads are grouped using feature-derived statistics, organized hierarchically, and given causal local windows while keeping the standard multi-head attention output interface. In 4096-token seed-0 experiments, several AAH-style variants achieved lower validation loss than pure full attention. However, the paper also notes that fixed/local controls could be competitive with adaptive hierarchy, and that stable local allocation and head-window assignment structure matter. The authors frame AAH as a structured context-allocation mechanism and report an Attention Coverage Ratio as a routing diagnostic.