Context-Ready Transformer beats 12-layer models with 1.7x speedup

Researchers have unveiled the Context-Ready Transformer, a novel architecture that blends transformer blocks with recurrent mechanisms to achieve deep model performance at a fraction of the cost. Author Mahesh Godavarti proposes adding a correction network that combines the cached output from the previous position (a summary of past context) with the current token embedding before it enters the transformer block. This pre-contextualization turns the architecture into a recurrent neural network during sequential inference, while training remains parallel by unrolling the correction process K times over the full sequence. The approach allows a 5-layer context-ready model to match a 12-layer standard transformer and generate 1.7× faster on an Nvidia A100 GPU. Even more striking, a single-layer model (D=1) with K=10 correction steps beats a 6-layer standard transformer with a 2.6× inference speedup, while maintaining perplexity within 0.01 of the parallel K=10 mode.

The architecture's design proves especially effective for wide representations and long contexts. In a pointer-chasing task that requires tracking variable references across multiple composition levels, the single-layer context-ready model solved all 10 levels, where standard transformers exhibited staircase-like depth dependence (deeper layers needed for more levels). The model also supports conversion from pretrained transformers by adding a zero-initialized correction feedforward network and fine-tuning, making it practical to upgrade existing deployments. This work challenges the prevailing notion that transformer performance scales strictly with depth, suggesting that smart recurrence can substitute for layer count, leading to faster, more memory-efficient inference without sacrificing quality.