Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantization

A new research paper by Takato Yasuno provides a comprehensive, three-stage blueprint for creating efficient, domain-specific Japanese language models that can run on consumer-grade hardware. The study systematically tackles the core challenges of training scale, base model selection, and quantization. In Stage 1, experiments identified 4,000 training samples as the optimal scale before overfitting occurs, with a test-set negative log-likelihood (NLL) reaching a minimum of 1.127. Stage 2 compared four Japanese LLMs, finding that Llama-3-based models with Japanese continual pre-training, specifically Swallow-8B and ELYZA-JP-8B, consistently outperformed multilingual alternatives like Qwen2.5-7B.

Stage 3 investigated architecture-aware quantization, a critical step for deployment. The research discovered that Llama-3 architectures improved under Q4_K_M 4-bit quantization, while models with Grouped-Query Attention (GQA), like Qwen2.5, degraded significantly, losing 0.280 points in performance. The final production recommendation is Swallow-8B quantized with Q4_K_M, which achieves a strong performance score of 2.83 out of 3, answers questions in 8.9 seconds, and occupies a compact 4.9 GB of memory. This methodology is designed to generalize to other low-resource technical domains, offering practitioners a clear, data-driven path to developing compact, high-performance specialist AI without requiring expensive infrastructure.