Perturbation is All You Need for Extrapolating Language Models

A new paper from statisticians Zetai Cen, Jin Zhu, Xinwei Shen, and Chengchun Shi introduces “Perturbation is All You Need for Extrapolating Language Models.” The core idea is simple: instead of training LLMs to predict the next token from an exact prefix, they first perturb the prefix to a semantically similar variant, then condition on that perturbed variant. This shifts the model from pure autoregressive next-token prediction to a hierarchical structure with a pre-post-additive noise layer. The framework allows the authors to develop a rigorous mathematical theory of extrapolability—the ability to make reliable predictions on token sequences outside the empirical support of the training corpus.

The paper includes both synthetic and real-world language experiments. The perturbation method consistently improves out-of-support (OOS) prediction accuracy while maintaining competitive in-support performance. Importantly, the approach does not require massive architectural changes; it works with existing transformer backbones and can be combined with standard training pipelines. The authors provide formal guarantees for when extrapolation is possible, making this one of the first theoretical treatments of a problem that has largely been addressed empirically.

For AI practitioners, this work suggests a practical route to making LLMs more robust to novel or rare inputs—a critical capability for real-world deployment. The 44-page paper (arXiv:2605.04344) is available with full proofs and code links. If the results scale, we might see perturbation-based training become a standard technique in future language model development, especially for use cases requiring high confidence on out-of-distribution data.