Study: Continual Pre-Training Beats Training from Scratch for SE LMs

A new arXiv study by Fabian C. Peña and Steffen Herbold investigates how to best adapt language models (LMs) to software engineering (SE) texts, such as issue reports, commit messages, and developer discussions. The researchers compared two approaches: continual pre-training (CPT) from an existing general LM and pre-training from scratch (PTS) on a new SE corpus they created. To ensure fairness, they controlled for token count and compute budget across model families and sizes. They evaluated both domain-specific adaptation using the SELU benchmark and general-language understanding using SuperGLUE.

Results show that reusing an existing LM via CPT dominates training a domain-native one from scratch. CPT yields only small and mostly statistically insignificant gains on domain tasks while leaving general-language understanding essentially intact. In contrast, PTS incurs a large and consistent penalty on both domain and general understanding. The only exception is for very small LMs under a token-rich budget, where PTS becomes competitive. The authors distill these findings into practical guidelines for practitioners and release their corpus and pre-trained models in a replication kit. The paper highlights that the cost of domain adaptation from scratch is rarely justified, encouraging engineers to leverage existing LMs with targeted fine-tuning instead.