NeurIPS/ICML 2025 Position Tracks Critique-Heavy, Lacking Agenda-Setters, Study Finds

A new arXiv paper by Fan Yang, Wenkai Li, and Jun Liu takes a hard look at the NeurIPS and ICML 2025 Position Paper Tracks—venues designed for agenda-setting research. The authors audited every publicly accessible submission to both tracks using a pre-specified rubric, then compared the submission mix against a reference class of historically field-defining ML papers, including AlexNet, the Transformer, and Concrete Problems in AI Safety. Their central finding: roughly 75% of audited position papers engage in reformist critique—attacking existing benchmarks, evaluation metrics, or methodologies—rather than introducing artifacts that give the community something new to build on, test, or contest.

The paper also finds that while these critique papers score well on 'artifact-coupling' (how closely they tie claims to a specific existing artifact), evidentiary depth does not predict reviewer ratings. The reference class of agenda-shifting works, by contrast, typically provides a new measurement protocol, benchmark proposal, toy implementation, dataset card, audit template, or falsifiable experimental program—things that actively reshape what research questions are possible. The authors close with four CFP-level interventions aimed at broadening the submission mix: explicitly soliciting direction-setting work, adding reviewer guidance for artifact proposals, offering dedicated submission categories for new benchmarks or datasets, and encouraging co-submission with implementation artifacts. These changes are designed to complement, not replace, the track's existing strength in hosting rigorous critique—ensuring position tracks don't become echo chambers of criticism without constructive new directions.