Aurora-80K: 80K-parameter LLM crushes tiny model expectations
80K-parameter model achieves 52% on BLiMP with 4,096-token vocab
Community researcher /u/Tall_Abrocoma_3533 has released Aurora-80K, a streamlined language model with only 80,000 parameters but surprisingly strong benchmarks. Despite its tiny footprint, it uses a factorized vocabulary of 4,096 tokens to maintain expressivity without bloating compute needs.
The model achieves 52.31% accuracy on BLiMP (a linguistic minimal pairs benchmark), 26.05% on Arc-Easy (elementary science reasoning), and 3.2902 bits-per-byte on Wikitext-2 (a text compression metric). These results suggest Aurora-80K defies the common trade-off between parameter count and performance in small models.
Aurora-80K is open-sourced and available on Hugging Face, where users can download and fine-tune it for tasks like text generation or classification without heavy infrastructure. It’s positioned as a research tool for exploring efficiency in LLMs, especially for edge or low-resource environments.
- Aurora-80K has only 80K parameters but uses a factorized 4,096-token vocabulary
- Benchmarks: BLiMP 52.31%, Arc-Easy 26.05%, Wikitext-2 3.2902 BPB
- Open-sourced on Hugging Face for research and fine-tuning
Why It Matters
Proves tiny models can deliver meaningful performance, unlocking AI for low-power devices and rapid prototyping.