Bilibili's Index-1.9B SLM rivals models several times its size

Bilibili, the Chinese video platform, has released Index-1.9B, a series of four open-source small language models (SLMs) detailed in their technical report. The series includes Index-1.9B-Base (foundation model), Index-1.9B-Pure (instruction-free control variant), Index-1.9B-Chat (aligned via SFT and DPO), and Index-1.9B-Character (augmented with retrieval-augmented generation for few-shot role-playing). Pre-trained on 2.8 trillion predominantly Chinese and English tokens, these models boast 1.9 billion non-embedding parameters. On standard benchmarks covering examination, reasoning, mathematics, and code, the Base model achieves an average score of 64.92, competitive with open models several times its size. The training leverages a novel Warmup-Stable-Decay learning-rate schedule, which concentrates curated data during the decay phase, and a Norm-Head output layer that stabilizes training under large learning rates.

The report also presents controlled studies on model depth, learning-rate magnitude and scheduling, the interaction between decay and data quality, and the effect of including instruction data during pre-training. Notably, the authors document an unexplained surge in benchmark performance midway through the constant-learning-rate phase. The Index-1.9B-Character model stands out for enabling customizable role-playing via retrieval-augmented generation, making it suitable for interactive applications. All four models, along with evaluation code, are released openly, providing researchers and developers with efficient, high-performing SLMs that can run on consumer hardware. This work underscores the growing trend of building capable small models that challenge the assumption that bigger is always better.