NE-BERT cuts perplexity 15.97x for 9 low-resource Indian languages

NE-BERT, a domain-specific multilingual encoder from researcher Badal Nyalang, tackles critical underrepresentation in NLP for Northeast Indian languages. The model was trained on approximately 8.3 million sentences covering 9 low-resource languages (including Pnar with only 1,002 sentences and Kokborok with 2,463) plus two anchor languages, Hindi and English. Using weighted data sampling and a custom SentencePiece Unigram tokenizer, it aggressively upsamples extremely low-resource languages to address vocabulary fragmentation—a key challenge that often causes existing models to produce poor results for these communities.

In evaluation, NE-BERT delivers substantial gains: 15.97x lower average perplexity than IndicBERT-V2, 7.64x lower than MuRIL, and 1.50x better tokenization fertility than mBERT. Downstream testing on part-of-speech tagging for three NE Indian languages confirms practical utility beyond perplexity metrics. Nyalang released NE-BERT, test sets, and the full training corpus under the permissive CC-BY-4.0 license, enabling researchers and developers to build on this work for digital inclusion and language technology in a linguistically diverse region that mainstream multilingual models have long ignored.