Fractale-350M-base replaces long context with fast-weight memory bank

Solo researcher kkuette has open-sourced Fractale-350M-base, a 386M-parameter base model pretrained from scratch on 10B tokens of code and English web text. The core innovation replaces traditional long-context mechanisms (like attention over past tokens) with a learned fast-weight memory bank. The model writes one gist vector per 512-token chunk into a bank of 8 slots, evicting the oldest (FIFO). Each slot expands via a hypernetwork into a small low-rank MLP that the token stream passes through. This makes memory part of the forward pass—no retrieval, no attention over stored text.

Measured performance is striking: GAP (cross-entropy difference between reset and carried bank) reached +9.4 nats on code and +7.3 nats on web, with the gap widening during training. At smaller scales, memory persists for 2000+ steps under eviction, and a single 13-token presentation can install a never-trained rule at 0.79–1.00 accuracy on unseen queries, outperforming test-time training by 138x in cost. The model is a base model only—no instruction tuning, no chat capabilities—but demonstrates a fundamentally different approach to memory in neural networks.