CrowdMath Dataset Reveals AI's Blind Spot in Collaborative Math Research

Researchers from MIT and the Art of Problem Solving (AoPS) have released CrowdMath, a novel dataset designed to test AI's ability to understand collaborative mathematical research. The dataset comprises 164 expert-annotated progress chains drawn from the MIT PRIMES-AoPS CrowdMath program (2016-2025), a long-running initiative where students and mathematicians work together on open problems, often leading to peer-reviewed publications. Each chain meticulously tracks a multi-participant forum discussion from the initial problem statement to a completed proof, with individual posts labeled by their functional role in the evolving solution process—categories include partial progress, proof completion, erroneous reasoning, and error identification. This goes far beyond typical math benchmarks that focus on well-specified problems with final answers or step-by-step solutions.

To probe the limits of current AI systems, the team benchmarked six frontier large language models on two key tasks: next-post prediction and post-role classification. The models performed surprisingly well on the first task, achieving 83-88% accuracy in predicting the next post in a discussion, indicating they can follow the local flow of mathematical dialogue. However, they struggled significantly on the second task: the best model achieved only a 0.42 macro-F1 score on classifying the functional significance of individual contributions. This dramatic performance gap reveals that while LLMs can mimic the surface-level structure of mathematical conversations, they fail to grasp the deeper collaborative reasoning—the back-and-forth of proposing partial arguments, identifying gaps, repairing errors, and synthesizing incremental steps into a coherent proof. CrowdMath exposes a critical blind spot in current AI for understanding how mathematical knowledge is actually built through human collaboration.