Dravidian GPT-2 models outperform multilingual AI
New monolingual GPT-2 models beat mGPT on Tamil, Telugu, Kannada, and Malayalam tasks
Researcher Venkata Naga Sai Vishnu Rohit Pulipaka trained five GPT-2 architecture models from scratch to evaluate Dravidian language capabilities—four monolingual models (one each for Tamil, Telugu, Kannada, and Malayalam) and one multilingual model. Each monolingual model used a dedicated 32K-vocabulary subword tokenizer, while the multilingual model shared a 64K-vocabulary tokenizer across all four languages. All models were trained on cleaned datasets including CC-100, Wikipedia, and Samanantar.
Results showed the monolingual models outperforming mGPT in sentiment classification and named entity recognition, while also demonstrating higher tokenizer efficiency across all tested languages. The study suggests dedicated monolingual architectures may be more effective than multilingual approaches for low-resource languages like Dravidian, challenging assumptions about shared multilingual training.
- Five GPT-2 models trained: 4 monolingual (32K vocab each) + 1 multilingual (64K shared vocab)
- Monolingual models beat mGPT on sentiment/NER tasks while using fewer tokens
- Datasets: CC-100, Wikipedia, Samanantar (cleaned) for all models
Why It Matters
Proves monolingual models can outperform multilingual ones for low-resource languages, improving AI for 250M+ Dravidian speakers.