KSR framework: Claude Sonnet 4 and GPT-5 lead LLM evidence synthesis tasks

A new arXiv paper from Wafa Shafqat, Mark Patterson, and Steven N. Liss tackles a growing problem: evidence in fast-moving fields is scattered across academic papers, industry reports, policy docs, and media, making synthesis painfully slow. The team built the Knowledge Synthesis Review (KSR) framework, a human-in-the-loop system that breaks evidence synthesis into four distinct cognitive tasks—screening, extraction, analysis, and synthesis—then benchmarks LLM-based systems on each task against expert reference standards. The framework automatically routes each task to the best-performing model while keeping expert validation in the loop.

They tested GPT-5, Claude Sonnet 4, Gemini 2.5 Pro, and NotebookLM on a 244-document subset drawn from a 1,893-document corpus on AI and work, spanning four source types. Against a gold standard with 92.2% inter-rater agreement (kappa = 0.80), no system led on all tasks. Claude Sonnet 4 achieved the highest screening accuracy (82.8%), while GPT-5 achieved the highest recall (91.8%) but with lower specificity. Extraction exceeded 90% agreement for titles and sources but degraded on author and reference fields. Performance dropped most on interpretive analysis and cross-source synthesis, where expert judgment was essential. A contamination check on post-cutoff documents ruled out inflated results. Applied to the full corpus, the routed workflow surfaced cross-source asymmetries and blind spots—like worker well-being, small firms, and the Global South—that single-source synthesis would miss. KSR offers a transparent, auditable, model-agnostic way to govern LLM assistance in research synthesis while preserving human accountability.