Researchers' DMM debiases AI annotations without gold-standard labels

A new arXiv paper tackles a growing problem in AI-assisted research: when scholars use AI models to measure variables for downstream statistical analysis, even small annotation errors can produce substantially biased results and invalid confidence intervals—despite accuracy rates above 90%. Existing fixes like design-based supervised learning or prediction-powered inference require gold-standard labels, which are often too expensive or impractical to obtain for large text datasets. In "Debiased Inference for AI-Generated Data without Gold-Standard Labels," authors Naoki Egami and Sooahn Shin introduce DMM (Debiased inference with Multiple imperfect Measurements) to solve this without any gold-standard labels.

The DMM framework combines multiple error-prone AI measurements, assuming they are conditionally independent given the latent true label and observed unit-level features such as text embeddings. This allows unknown misclassification rates to vary across annotation methods (e.g., different LLMs) and across individual texts. Building on established CP decomposition results, the authors use semiparametric inference theory to prove that DMM's estimator is consistent and asymptotically normal, enabling valid inference for common social science analyses. Their simulations demonstrate that DMM produces valid inference and that incorporating additional accurate—though still imperfect—measurements improves efficiency. The paper also provides diagnostics to test the key conditional independence assumption, making the framework practical for LLM annotation applications.