How did ‘large’ language models get that way? The role of Transformers and Pretraining in GPT

In his LessWrong essay, Oliver Sourbut traces how large language models (LLMs) like GPT grew to their enormous scale. He argues the answer lies in the 'P' (pretrained) and 'T' (transformer) of GPT. Self-supervised learning—training a model to predict the next token on vast unlabeled internet text—provides the bulk of the 'cake,' as Yann LeCun famously analogized. This approach sidesteps the expense of curated, human-labeled datasets and forces the model to learn generalizable linguistic concepts. The Transformer architecture, with its attention mechanism, efficiently handles long-range dependencies in sequences, enabling models to process entire paragraphs or documents as context. This combination allowed scaling from millions to billions of parameters, driving the exponential growth in model size.

Sourbut notes that while this self-supervised pretraining was the foundation, the paradigm has evolved by 2025–2026, with reinforcement learning and supervised fine-tuning playing larger roles (the 'icing' and 'cherry'). The essay is part of a series examining implications for AI reasoning and explainability, hinting that current trends may sacrifice some of the transparency that emerged from this training regime. For professionals, understanding this history clarifies why LLMs are so data-hungry and compute-intensive, and why future shifts in training methodology could change their capabilities and interpretability.