OpenLanguageModel: Open-source library for readable, composable small LM pretraining
PyTorch library that turns model code into a readable architecture diagram with 90.6% scaling efficiency.
OpenLanguageModel (OLM) is a new open-source PyTorch library designed to make small language model pretraining transparent and composable, targeting education and research. Unlike black-box frameworks, OLM's code reads like the architecture itself, using ordinary modules like Block, Residual, Repeat, and Parallel to describe how components are wired. This allows models to move seamlessly from a teaching notebook to a complete pretraining run or a research ablation. The library connects the readable model layer to tokenizers, local and streaming datasets, optimization, mixed precision, callbacks, checkpoints, and hardware-aware execution across CPU, single-GPU, and single-node multi-GPU setups.
To demonstrate its utility, the authors trace GPT-2 from diagram to code, launch a FineWeb-Edu training script, replace one attention component, and let AutoTrainer configure the available machine. OLM ships with 27 pretrained presets across nine familiar model families, alongside documentation that progresses from LM fundamentals to architecture research. Validation shows close agreement with independent reference implementations, 90.6% four-GPU weak-scaling efficiency for a 348M-parameter workload, compact architecture edits, and positive early usability results. Released under the MIT license, OLM is available via PyPI, GitHub, and its documentation site.
- Uses modular primitives (Block, Residual, Repeat, Parallel) to make model architecture explicit and code readable.
- Achieves 90.6% weak-scaling efficiency on 4 GPUs for a 348M-parameter model during pretraining.
- Includes 27 presets across 9 model families, with support from notebooks to production-scale runs.
Why It Matters
Democratizes LLM pretraining by making internal mechanics visible and composable for education and research.