Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation

Researcher Joseph James has published a seminal paper titled 'Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation' on arXiv. The work addresses a foundational yet increasingly complex problem in AI development: reliably measuring agreement between human annotators who label data for training and evaluating models like Claude 3.5 or GPT-4o. As NLP tasks expand beyond simple categorization to include segmentation, subjective judgment, and continuous ratings, choosing the correct statistical metric—be it Cohen's Kappa, Krippendorff's Alpha, or Intraclass Correlation—has become critical for ensuring dataset quality and reproducible research.

The paper systematically organizes agreement measures by task type and delves into how practical issues like label imbalance and missing data can skew reliability estimates. It moves beyond mere description to prescribe best practices for transparent reporting, advocating for the use of confidence intervals and detailed analysis of disagreement patterns. This framework is designed to help teams at companies like OpenAI, Anthropic, and Meta build more consistent and interpretable benchmarks, directly impacting the reliability of the AI models that depend on this human-curated data.