Just did an analysis on ICLR 2025 vs 2026 scores and WOW [D]
Analysis reveals reviewer consensus at top AI conference has significantly worsened year-over-year.
A viral analysis of peer review data from the International Conference on Learning Representations (ICLR) reveals a significant deterioration in consensus among human reviewers between the 2025 and 2026 cycles. The analysis, based on data fetched from OpenReview, shows that while the correlation between two reviewers was about 0.41 for ICLR 2025, the 2026 data indicates a much lower level of agreement. The key metric—the mean standard deviation of scores assigned by different reviewers to the same paper—jumped from 1.186 in 2025 to 1.523 in 2026, a 28% increase. This suggests that for a given paper, reviewers' opinions are diverging more sharply, making the final acceptance outcome increasingly unpredictable.
Researchers have long suspected that peer review at elite AI conferences can be inconsistent, but the quantitative evidence from this analysis is striking. The findings imply that for many papers submitted to ICLR 2026, whether a submission is accepted or rejected may depend more on which reviewers are assigned than on the paper's intrinsic quality. This "lottery" effect raises serious concerns about the fairness of the process and the potential for high-quality work to be overlooked. As AI research accelerates, the community is forced to confront whether its primary quality-control mechanism is robust enough to handle the volume and complexity of modern submissions.
- Reviewer score correlation for ICLR 2026 is significantly lower than the 0.41 measured for ICLR 2025.
- The standard deviation of scores within the same paper increased 28% year-over-year, from 1.186 to 1.523.
- The data suggests the peer review process at top AI conferences is becoming increasingly inconsistent and lottery-like.
Why It Matters
This calls into question the fairness and reliability of how groundbreaking AI research is selected and published, impacting careers and scientific progress.