LPC-SM: Local Predictive Coding and Sparse Memory for Long-Context Language Modeling
A new 158M-parameter model separates memory and attention, improving long-context tasks with a 12% loss reduction.
Researcher Keqin Xie has introduced LPC-SM (Local Predictive Coding and Sparse Memory), a novel hybrid architecture designed to tackle the challenges of long-context language modeling. The core innovation is a clear separation of duties within the model block: local attention handles immediate token interactions, a persistent sparse memory stores long-range information, a predictive correction mechanism refines outputs, and a runtime control system manages the flow. This division is governed by a novel technique called Orthogonal Novelty Transport (ONT), which controls what information gets written to the slow, persistent memory. The approach fundamentally challenges the prevailing paradigm where Transformer-based models use self-attention to manage both local and long-range dependencies simultaneously.
The paper evaluates a relatively compact 158M-parameter model across three progressive stages. The results demonstrate the efficacy of this decomposed design. For instance, removing a key memory component (mHC) worsened the language modeling loss, while implementing adaptive sparse control improved the final loss in a continuation task from 12.137 to 10.787. Crucially, the full LPC-SM system remained stable when processing full 4096-token sequences, a critical test for long-context models. It also showed a significant improvement on a 'delayed-identifier' diagnostic, reducing the cross-entropy loss from 14.396 to 12.031. These findings provide concrete evidence that organizing autoregressive modeling around specialized components—rather than a monolithic attention mechanism—can lead to more efficient and effective handling of long sequences.
- Proposes a 158M-parameter hybrid architecture that separates local attention, persistent memory, and control mechanisms.
- Uses Orthogonal Novelty Transport (ONT) to govern sparse memory writes, improving long-range information retention.
- Shows a 12% improvement on a key diagnostic and stable performance on 4096-token sequences versus attention-only baselines.
Why It Matters
This research could lead to more efficient, scalable AI models that process long documents, codebases, and conversations with less computational cost.