Reddit user scales Gemma 4 to 40.5B with novel layer stacking
After first attempt flopped, new technique blends neighboring layers' weights to revive dead layers.
A Reddit user named TOTORONG has achieved a notable open-source AI milestone by extending Google's Gemma 4 model from roughly 31 billion parameters to 40.5 billion parameters. The approach involved stacking additional transformer layers (88 layers total) into the pre-trained model. The first attempt failed miserably—new layers initialized to zero (a common 'no-op' trick) remained dead weight because they received no gradient signal during backpropagation. The model simply flowed around them, and they never learned.
The successful follow-up introduced three key changes. First, instead of zero-initialization, each new layer was initialized as a weighted blend of the two neighboring layers' weights, giving it immediate functional behavior and thus gradient flow. Second, insertion points were carefully chosen to avoid critical 'manager' positions (like attention layers with strong skip connections). Third, TOTORONG left the per-layer scaling factors (RMSNorm gains) neutral instead of averaging them, preserving signal integrity. Training then proceeded in two phases: first freezing all original layers and training only the inserted ones on a small dataset of STEM reasoning and Korean legal Q&A, then fine-tuning the entire model. The result is a working 40.5B parameter model available on Hugging Face, demonstrating a practical method for scaling open-source LLMs without full retraining.
- Zero-initialization of inserted layers caused them to receive zero gradient signal, making them dead weights.
- Successful method initializes each new layer as a blend of its two neighboring layers' weights for immediate gradient flow.
- Two-phase training: first freeze original model and train only new layers, then full fine-tuning.
Why It Matters
Provides a practical technique for scaling open-source LLMs without expensive retraining, lowering barriers for AI research.