BCMT: new blockwise memory transformer rivals dense attention with faster training

Transformer architectures dominate language modeling but suffer from quadratic attention complexity as sequence length grows. Rachid Arezki's new paper introduces BCMT (Blockwise Causal Memory Transformer), which sidesteps this bottleneck by treating local and global context separately. Within each block, dense causal self-attention handles token interactions; across blocks, an adaptive summary is computed and aggregated through an exponential causal memory. This memory is then injected back into token representations, propagating long-range information without explicit global attention or learned memory states. The design remains fully parallelizable and compatible with standard dense attention implementations.

Experiments on language modeling with context lengths up to 1024 tokens show BCMT achieves validation performance comparable to Dense Transformers while significantly improving training throughput and reducing memory consumption. An ablation study confirms these gains come from the proposed memory mechanism. Unlike recurrent memory architectures that rely on learned hidden states, BCMT's memory is derived directly from block summaries, making it both simpler and more parallel-friendly. These results suggest exponential causal memory built from block summaries is a viable alternative to dense global attention for long-context modeling, with potential to scale to even longer sequences.