CARVE: New Attention Model Boosts Efficiency 19% with Smarter Erase Gate

Current recurrent models like GDN-2 suffer from memory-blind gating: the erase gate decides what to forget without consulting the stored memory. This leads to three coupled defects: wasted parameters on the value-axis erase mask, and a mathematical blocker for the WY-form triangular chunk solver that makes recurrent training competitive with Transformers. CARVE (Content-Aware Recurrent with Value Efficiency) resolves all three through a single principle: erase only on the key axis. This not only proves necessary and sufficient for the WY-form solver but also reuses the recurrent output tensor as a free content signal, and replaces the per-value write-gate projection with a single scalar per head. At initialization, CARVE is bit-identical to GDN-2, so any quality gains come purely from what the content gate learns.

At 1.3B parameters trained on 100B tokens, CARVE achieves WikiText perplexity 15.72 — a 0.18 improvement over GDN-2 representing a 4.5-sigma effect. It outperforms every recurrent baseline on nine commonsense reasoning benchmarks and sets state-of-the-art on all RULER retrieval probes. These gains come at just 0.4% throughput overhead, with 13% lower peak memory usage and 19% fewer parameters. The paper also presents six formal theorems covering memory capacity, Lyapunov stability, gradient flow, expressivity separation, Pareto-optimal chunk size, and hybrid optimality.