Quantifying and Attributing Polarization to Annotator Groups

Researchers have created a new metric to measure polarization between different groups of people labeling online content. It works even when groups are of very different sizes. When applied to hate speech and toxicity datasets, they found strong, persistent disagreement linked to annotator race. Religious and less educated annotators also showed distinct patterns. The team provides an open-source library and estimates the minimum number of annotators needed for reliable results.