Finance-grade LLMs: Point-in-time models rival unrestricted ones
No more lookahead bias: 4B-parameter models trained on date-stamped data only.
Large language models trained on unrestricted internet data inevitably learn from future information, creating lookahead bias that invalidates backtests in finance and causal inference in social sciences. To solve this, Bryan Kelly, Semyon Malamud, Johannes Schwab, and Teng Andrea Xu introduce point-in-time LLMs trained exclusively on text available up to each specific date. By scaling decoder-only transformers to 4 billion parameters and pre-training on 1 trillion tokens from FineWeb with careful chronological filtering, they construct a sequence of monthly model checkpoints spanning 2013–2024.
On standard benchmarks for common-sense reasoning and language understanding, their point-in-time models approach the performance of leading open-weight models like Gemma-3-4B and LLaMA-7B trained on unrestricted data—though a gap remains on several tasks. Instruction fine-tuning via LoRA further improves downstream usability. The team releases the full pipeline, including dataset construction, training infrastructure, and evaluation code, enabling reproducible temporal language modeling and rigorous causal inference without future data leakage.
- Eliminates lookahead bias by training only on text available before each calendar date
- Scales to 4B parameters trained on 1 trillion chronologically filtered tokens
- Monthly checkpoints from 2013-2024 approach performance of Gemma-3-4B and LLaMA-7B on reasoning tasks
Why It Matters
Enables trustworthy backtesting and causal inference in finance and social sciences with near-state-of-the-art LLMs.