LongNovel benchmark reveals LLM hallucination patterns in long-form summarization

LongNovel, a new benchmark from Ruizhi Zhang and collaborators, tackles a persistent problem in AI: hallucination in long-context summarization. While modern LLMs boast massive context windows, they still fabricate details when summarizing lengthy documents. The researchers argue novels are the ideal test case—unlike news or academic papers, novels contain rich event sequences and dialogue that stress a model's ability to track narrative details. LongNovel is built from 29 Chinese novels spanning 16k to 100k tokens, alongside chapter-level data from the BookSum dataset, making it a bilingual (Chinese and English) multi-scale benchmark. It categorizes hallucinations into 8 distinct types, from entity errors to timeline inconsistencies, providing a nuanced framework for evaluation.

To ensure data quality, the team combined Multi-Model Arbitration and Entity-Referenced Hallucination Generation, which uses multiple LLMs to generate and verify hallucinated content while cross-referencing entities to maintain authenticity. The test set was manually revised to eliminate noise and guarantee a balanced distribution across hallucination categories. Extensive experiments show that LongNovel is genuinely difficult for current models, exposing weaknesses that simpler benchmarks miss. For developers and enterprises deploying long-context summarization tools, LongNovel offers a rigorous, granular way to measure reliability—crucial as AI moves into book summarization, legal document analysis, and other high-stakes long-form applications. The dataset is publicly released to support future research.