New Math Fix Makes AI Ratings and Peer Reviews More Honest

Researchers propose a simple ordinal peer-prediction mechanism for settings such as conference peer review and Reinforcement Learning from Human Feedback for large language models, where there is no objective ground truth against which reports can be directly verified. Existing peer-prediction methods do not provide incentives for agents to truthfully compress rich private information—such as reviews—into coarser ordinal reports—such as rankings—requested by the mechanisms.

The proposed mechanism rewards agents based on pairwise agreement, measured by Kendall tau. Under a broad class of Bayesian Stochastically Order Consistent models—including classical models such as Mallows and Thurstone—the authors prove that truthful reporting is stochastically dominant, and strongly truthful under index-equivariant strategies.

This provides robust incentives even for agents with non-linear utility functions, such as risk aversion. The mechanism further accommodates heterogeneous agents, top-K reports, and subset assignments, where each agent is scored against one randomly drawn peer. Experiments on synthetic Mallows and Thurstone data, together with a real-data stress test on the SUSHI preference dataset, show right-shifted truthful-payment distributions and the highest normalized sensitivity among tested baselines.