Independent dev trains 1.1B-parameter LLM for $200
One dev spent $200 to train a 1.1B LLM from scratch on 20B tokens...
Independent developer Nicola S successfully trained a 1.1B-parameter language model named **gemmeh** from scratch using 20 billion tokens of educational data from FineWeb-edu, all for approximately $200 in compute costs. The project began as a personal learning exercise but evolved into a fully functional open-source model. The base model was fine-tuned using LoRA (Low-Rank Adaptation) on the OpenHermes dataset to create an instruction-following chat variant.
The training process was conducted on Vast.ai’s cloud GPUs, with the developer experimenting with three model sizes (185M, 527M, and 1.1B parameters) before settling on the final 1.1B configuration. Training the 1.1B model on 20B tokens took 130 hours on an H100 GPU, achieving a final validation perplexity of 10.93. The architecture was based on Google’s Gemma3 but included adjustments such as a smaller 32K vocabulary, a 4096-token context window (without sliding window attention), and custom hyperparameters to optimize performance.
- Trained a 1.1B-parameter LLM (gemmeh) from scratch on 20B tokens of FineWeb-edu data for ~$200 using Vast.ai GPUs.
- Used LoRA fine-tuning on OpenHermes to create an instruction-tuned chat model, with all assets open-sourced on Hugging Face.
- Achieved a final validation perplexity of 10.93 after 130 hours of training on an H100 GPU, with a custom 32K-token vocabulary and 4096-token context.
Why It Matters
Proves large-scale LLM training is accessible to solo developers with modest budgets, accelerating open-source AI innovation.