PersonaJudge simulates individual AI evaluators with 9.9% accuracy boost

PersonaJudge introduces a novel simulation approach for AI evaluation that moves beyond consensus preferences to model individual human evaluators. By combining categorical judgments with evaluator-specific auxiliary data—such as retrospective reasoning traces and interface telemetry—the method enables LLMs to simulate individual annotators via in-context learning. The study leverages a 4×4×4 factorial design with 32 trained annotators across 4,200 preference judgments.

Key findings show that reasoning traces provide the largest gains (up to 9.9 percentage points over the base judge), while interface telemetry often hurts performance despite lower collection costs. Researchers also discovered that simulation difficulty is systematic—predicted by an evaluator's neutral usage (especially on Helpfulness) and divergence from consensus. The neutral-usage tendency itself was found to be a cross-task-stable property (r = 0.728), offering methodological insights for scaling individual-aware AI assessment.