AI Can Now Grade Its Own Work Without a Second Opinion
Could make checking AI answers — and paying data labelers — cheaper and fairer.
A new article introduces mutual evaluation of a replicable task worker and a critic, with both modeled as strategic agents. The critic chooses a finite-valued rule that induces an evaluation score on joint report laws, and their common payoff is analyzed through regret relative to the unrestricted critic envelope. The critic rule is distinct from the evaluation score. This class enables a peer-free information elicitation mechanism using conditionally independent replications of a worker on the same task — a replication-loop mechanism that implements a type-agreement payoff using same-task replications and new-task samples. In contrast to the peer-prediction and scoring-rule literature, the article shows implementations that produce unbiased Pearson and Shannon information scores without requiring peers, a ground-truth reference, or likelihood-ratio estimation. A valid binary critic can also be represented by shared finite type annotations of worker returns. One runtime restriction: the number of required replicas is random and can depend on the critic rule. Other timing effects, such as commitment and reoptimization, yield distinct incentives, connecting the framework to variational peer prediction. The mechanism class illustrates why strategic considerations matter for both critic and worker agents.
- The idea: repeat the same task several times instead of comparing workers to a crowd, so no peers or answer key are needed.
- It reproduces two standard trust scores (Pearson and Shannon information) by pure math, with a computer-verified proof.
- It's theory only — no tests, no product, and the number of repeats needed can be random and unpredictable.
Why It Matters
Could make AI training and quality checks cheaper and harder to game — if it ever works in practice.