LoKiFormer cuts LLM training time by 1.33x with new architecture
Researchers propose LoKiFormer, a new LLM architecture that speeds up training by 1.33x while improving efficiency
Researchers from multiple institutions (led by Qiuwu Chen) have proposed LoKiFormer, a groundbreaking LLM architecture designed to address inefficiencies in traditional transformer-based models. The team identified two key limitations in current architectures: (1) self-attention's lack of explicit locality bias, leading to redundant modeling of local patterns, and (2) mixture-of-experts (MoE) systems coupling knowledge storage with computation paths, restricting flexible knowledge access.
To solve these challenges, LoKiFormer introduces two novel components: the Local Fusion Attention (LFA) module, which integrates convolutional fusion into attention mechanisms to explicitly capture local patterns, and the Knowledge Memory Module (KMM), which decouples global knowledge storage from computation via parametric key-value memory. Together, these innovations enable LoKiFormer to achieve 1.33x faster convergence during pretraining while improving model efficiency. The architecture was accepted to ICML 2026 and demonstrated superior performance over existing models in experimental results.
- LoKiFormer accelerates LLM pretraining by 1.33x compared to baseline models
- Introduces Local Fusion Attention (LFA) for explicit local pattern capture and Knowledge Memory Module (KMM) for decoupled knowledge storage
- Accepted to ICML 2026; outperforms traditional attention and MoE architectures
Why It Matters
Could slash training costs and time-to-market for new LLMs by over 30%, making advanced AI more accessible