Reducing Detail Hallucinations in Long-Context Regulatory Understanding via Targeted Preference Optimization

Researchers from multiple institutions introduced DetailDPO, a targeted preference optimization framework that tackles the persistent problem of detail hallucinations in large language models (LLMs) when processing long regulatory documents. These hallucinations are subtle but critical errors in threshold values, units, scopes, obligation levels, and conditions that maintain surface plausibility while corrupting safety-critical parameters. The team first formalized this phenomenon through a fine-grained Detail Error Taxonomy of five distinct error types, then built DetailBench, a benchmark comprising 172 real regulatory documents and 150 synthetic documents spanning three jurisdictions, with human-annotated detail-level ground truth totaling 13,000 preference pairs.

DetailDPO constructs contrastive pairs that differ in exactly one detail dimension, concentrating the DPO gradient signal on detail-bearing tokens. The researchers provided theoretical analysis showing why minimal detail perturbation pairs yield gradient concentration under mild assumptions. Experiments on the Qwen2.5 family (7B, 14B, 72B) and Llama-3.1-8B across three context-length tiers (8K-64K tokens) demonstrated that DetailDPO reduces the Detail Error Rate by 42-61% relative to baselines, with consistent gains across all five error types and cross-domain transfer to financial and medical documents. This work has significant implications for regulated industries like finance, healthcare, and legal where precise reading of lengthy documents is critical.