Mistral model achieves 0.67 F1 on Voltaire indexing task

Thematic indexing—assigning structured conceptual labels to sections of text—is essential for navigating massive literary and historical editions, but it remains a slow, manual process. This paper from Arana-Catania, Pink, and Roe applies machine learning to automate the task, using two sub-corpora of the Complete Works of Voltaire: the 'Essai sur les mœurs' and 'Questions sur l'Encyclopédie.' They frame the problem as multi-label classification, where a model must predict the exact set of index entries a professional indexer would assign to each page. The team compared a wide range of approaches, from traditional encoder-based models with classification heads to generative LLMs fine-tuned via Low-Rank Adaptation (LoRA), spanning model sizes from roughly 3 to 120 billion parameters.

The study's standout performer was a 4-bit quantized Mistral model, which achieved F1 scores of up to 0.67—a result the authors argue is a lower bound, given the subjectivity of human indexing and frequent cases where model predictions were semantically valid but differed from the original print index. They also evaluated cross-corpus generalization and conducted a detailed qualitative analysis of literary and rhetorical features that proved especially resistant to automation. The findings have significant implications for providing structured thematic access to large-scale historical and literary corpora, potentially transforming how scholars navigate archives of 18th-century works and beyond.