Tsinghua's new SurveyReview benchmark cuts AI evaluator errors by 39%

As LLMs turn survey writing into an automated process, evaluating the quality of those surveys has become the bottleneck. Off-the-shelf LLM-as-a-judge methods often diverge from what human reviewers expect. To fix this, a Tsinghua team created SurveyReview, a multi-dimensional benchmark built from 675 annotated survey papers and 1,630 structured peer-review reports. They converted free-form comments into four scoring dimensions (Readability, Criticalness, Comprehensiveness, Structure), each paired with supporting rationales, and released standardized train/test splits plus an evaluation protocol to quantify how well automatic evaluators align with human reviewers.

To validate the benchmark, the researchers developed SurveyAlign, a strong baseline evaluator created by fine-tuning Qwen3-32B with LoRA on the annotated data and augmenting it with external knowledge for knowledge-intensive dimensions. On the test set, SurveyAlign significantly outperformed prompt-based judging with GPT-5.2, reducing average MSE from 2.28 to 1.38 (a 39% drop) and MAE from 1.15 to 0.69 (a 40% drop) across all four dimensions. The work establishes the first reproducible, reviewer-aligned evaluation framework for survey reviewing, and both code and data are publicly available to support future research.