TimeCapsuleLLM: 160GB dataset of 1800s texts trains Victorian-era AI

A hobbyist AI developer spent a year curating and pre-training language models exclusively on 19th-century English texts, culminating in the TimeCapsuleLLM project. The dataset comprises 40 billion tokens (160GB) from 1800-1875, sourced from both England and the United States. To evaluate feasibility, the developer trained a 500M parameter model on a 5B token subset, then fine-tuned it on synthetic question-answer pairs automatically extracted from the historical corpus. This allows users to query the model about historical figures, places, and events, with notably better performance on London-centric topics. While the evaluation model lacks high factual accuracy, its outputs—such as a recipe for plum pudding—demonstrate coherent Victorian-era language patterns.

Despite the current limitations, the project signals potential for specialized historical LLMs. The developer plans to scale up to a 2B parameter model using the full 40B token dataset, which could produce more reliable and nuanced answers about 19th-century life, literature, and culture. TimeCapsellLLM is open-sourced on GitHub and Hugging Face (haykgrigorian/TimeCapsuleLLM-English-1800-1875-v3mini-eval1-500M). For researchers in digital humanities or historical linguistics, such domain-specific models offer a novel way to explore period-accurate language generation and retrieval.