Developer rebuilds Gemma 4 31b into 26b with Attention Residual Networks
Removing weakest SWA layer and adding Moonshot's attention residuals for better long context.
A developer is undertaking an ambitious project to rebuild Google's Gemma 4 31b model into a leaner, more capable 26b version. The first step involves running ablation tests to identify the weakest SWA (Sliding Window Attention) layer—Layer 3 consistently underperforms and will be removed. After removal, the attention windows will be rescaled to 1024/2048/4096/8.1K across the remaining SWA layers, plus a global layer per 'Block' design. To further improve global coherence, the developer plans to add Attention-based Residual Networks, a technique from Moonshot (paper early 2026), which should allow global layers to exchange information more fluidly.
Training will use TopK logit reconstruction from the original 31b model, freezing the top and bottom layers to keep tokenization and output stable while the internals adapt. The developer estimates feeding a few billion tokens from a yet-to-be-curated dataset, potentially using spot pricing on a B300 GPU. The rebuild starts from an abliterated core to ensure no censorship, and the thinking training will also be uncensored. The final model is expected to drop from 30.81B to 26.02B total parameters while improving long-context performance and global understanding. The project is open to dataset and compute donations.
- Removes weakest SWA layer (Layer 3) and rescales remaining attention windows to 1024/2048/4096/8.1K plus global layer.
- Adds Moonshot's Attention-based Residual Networks for better global coherence across layers.
- Uses TopK logit reconstruction from original 31b model to retrain internals, targeting 26.02B parameters with no censorship.
Why It Matters
A smaller, uncensored Gemma with improved long context could set a new standard for community-driven model rebuilding.