Hybrid human-AI markets outperform pure models for replication forecasts

The replicability crisis has long plagued empirical science—many published findings fail to hold up in subsequent studies. Existing assessment methods typically rely either on human expert panels (which suffer from cognitive biases and limited literature exposure) or on machine learning models trained on paper metadata (which often miss subtle contextual cues). A new paper from researchers at Penn State and partner institutions proposes a hybrid approach: a prediction market where both algorithmic agents and human participants trade contracts on whether a given finding will replicate.

The agents are trained on outcomes from hundreds of prior replication studies, giving them a statistical baseline, while human traders inject real-time domain knowledge and intuition. In multiple live experiments spanning different academic disciplines, the hybrid market matched or surpassed the accuracy of pure artificial prediction markets in nearly all cases. This suggests that combining machine learning with crowd wisdom can yield more reliable replication forecasts than either method alone. The work points toward a scalable, cost-effective way to triage scientific claims and prioritize replication efforts.