Hobbyist expands Gemma4-31B to 44B with 88 layers for niche domains
When Google stopped at 31B, a Reddit user scaled Gemma4 to 44B on their own hardware.
A Reddit user, Desperate-Sir-5088, has created extGemma4-44B, a dense 44-billion-parameter model by expanding Google's Gemma4-31B. Since Google hasn't released a larger dense Gemma4 variant, the developer used a two-stage expansion: first extending from 60 to 80 layers via identity-init (following the LLaMA Pro approach with a Gemma4-specific fix), then a second block duplication to 88 layers (~47B params) on top of the already fine-tuned model. The model was fine-tuned on Korean legal and STEM data, aiming to add new domain knowledge without overwriting existing weights. Early results for these specific use cases are promising, though tool-calling remains untested.
Looking ahead, the developer plans to explore further expansion (96–100 layers) or focus on data quality, and is seeking collaborators for coding and tool-calling datasets. They also aim to apply similar techniques to MoE architectures like GLM-5.2 or DeepSeek V4-Flash, and invite feedback from the community. The full writeup, including architecture details and training verification showing that duplicated attention layers contributed meaningfully, is available on Hugging Face.
- Expanded Gemma4-31B from 60 to 88 layers (~47B params) using identity-init and block duplication
- Fine-tuned on Korean legal and STEM data to add domain capacity without overwriting existing weights
- Developer seeks help with coding and tool-calling datasets, and plans MoE expansion experiments
Why It Matters
This shows how the open-source community can extend dense models for niche domains without waiting for big labs.