Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation
A massive new review exposes the fundamental flaw poisoning every AI model we use.
A systematic review of 346 papers from premier AI venues reveals the concept of 'ground truth' in data labeling is a harmful illusion. The study argues treating human disagreement as mere 'noise' ignores vital cultural and subjective signals, a problem worsened by model-mediated annotation and geographic bias. It found systemic failures where precarious data workers prioritize compliance over honest subjectivity, enforcing Western norms as universal benchmarks and removing authentic human perspectives from AI training loops.
Why It Matters
This challenges the core assumption behind all modern AI training, suggesting today's models are built on culturally biased, flawed foundations.