Hobbyist builds custom 25M parameter autocomplete model on 32GB VRAM
Training from scratch on 100M+ tokens for comedy or tech autocomplete.
A hobbyist developer on Reddit is diving into building a custom small language model from scratch, leveraging 32GB VRAM. Their plan: train a 25M parameter autocomplete model that predicts the next token, sentence, or paragraph, rather than generating full chat responses. The project is framed as a learning experience, not a production-grade foundation model.
Data is the primary bottleneck. The developer estimates needing on the order of 100M+ tokens for meaningful training—roughly 4x the parameter count. Two dataset directions are under consideration: a comedy model trained on cleaned YouTube transcripts to learn punchline patterns, or a technical model focusing on Python, Linux, or cybersecurity content. They're asking the community for high-quality sources beyond Wikipedia, Common Crawl, or synthetic data, and for tips on formatting data for autocomplete-style training versus chat or Q&A formats.
- Targets a 25M parameter model with 32GB VRAM, requiring roughly 100M+ training tokens
- Considers comedy transcripts for setup-to-punchline learning or technical data for Python/Linux/cybersecurity
- Seeks dataset recommendations beyond Wikipedia/Common Crawl and advice on autocomplete vs. chat data formatting
Why It Matters
Demonstrates growing accessibility for hobbyists to train custom small models on limited hardware.