PA-DSL: New method corrects noisy human labels in AI classification

A new paper from Robert Chew and Matthew R. Williams introduces PA-DSL (Partially Adjudicated Design-Based Supervised Learning), a method designed to tackle a common but often ignored problem: noisy human labels in the audit sets used to correct automated classifiers. When researchers use AI to label large datasets, they typically rely on a small human-annotated sample (audit set) to estimate and correct errors. But those human labels themselves can be noisy, and only a fraction are typically reviewed by experts (adjudicated). PA-DSL leverages the adjudicated cases to rectify the noisy human labels, then uses the corrected audit to debias analyses based on the full automated label set. The estimator is valid for a broad class of downstream analyses when the probabilities for audit selection and adjudication are known.

In experiments on synthetic data and the Wikipedia Detox dataset, PA-DSL maintained nominal coverage (the rate at which true values fall within confidence intervals) and reduced root-mean-square error (RMSE) by 10–17% compared to using only adjudicated labels. This is significant because many real-world ML pipelines discard noisy labels or treat them as ground truth. PA-DSL shows that when noisy human labels contain recoverable signal, incorporating them with partial adjudication can yield more accurate and reliable results without requiring full expert review of the audit set. The method is particularly relevant for social science, healthcare, and content moderation applications where expert annotation is expensive but automated classifiers need rigorous validation.