Auditing the Auditors: Does Community-based Moderation Get It Right?

A team of researchers from UC Berkeley and Microsoft Research has published a critical audit of X's (formerly Twitter) Community Notes system, a flagship example of crowd-sourced content moderation. The study, titled 'Auditing the Auditors,' focuses on a design flaw the authors term 'consensus-based auditing,' where users' eligibility to participate is tied to their agreement with the platform's eventual aggregated outcome. By analyzing data from after September 2022—when this auditing method was adopted—the researchers found clear evidence of 'strategic conformity.' Minority contributors' evaluations systematically drifted toward the majority view, and their participation share dropped on precisely the controversial topics where independent, diverse signals are most valuable.

Motivated by these findings, the researchers propose a novel two-stage algorithm to reform such systems. The method first accounts for inherent differences across content and contributors using a latent-factor model. In the second stage, it weights contributors not by their agreement with the consensus, but by the stability and predictability of their past 'residuals'—the part of their evaluations not explained by the common model. This means a user who consistently provides unique, informative context, even when disagreeing with the majority, gains greater influence. When tested on the Community Notes data, this approach improved out-of-sample predictive performance while successfully avoiding the penalization of disagreement that plagues the current system.