Manual annotation tasks teach AI students about data subjectivity and bias
Students who annotated skin lesions learned that AI labeling is subjective, not objective...
A recent study published on arXiv (arXiv:2607.20149) titled 'Data Annotations as Pedagogical Hints' investigates whether manual data annotation tasks can teach machine learning students about the subjectivity inherent in AI data. Researchers from Fontys University of Applied Sciences (Netherlands) and IT University of Copenhagen (Denmark) designed an annotation activity where 43 students labeled skin lesion images for hair coverage on a 3-point scale. Surveys measured understanding of annotation ambiguity, data quality, bias, fairness, implementation barriers, and pedagogical effectiveness.
The results showed that self-reported familiarity with course content substantially increased across all concepts. Most students recognized that personal interpretation affects annotations, and they rated the activity as more effective than traditional lectures for understanding bias. The main drawback was emotional unease from viewing medical images. Importantly, many students still requested clearer guidelines to reduce disagreement, suggesting they had not fully internalized that disagreement from different perspectives is a learning feature, not a bug. The authors recommend ensuring interpretive ambiguity in materials, reducing repetitive workload, mitigating emotional unease, and explicitly framing disagreement as a learning opportunity.
- 43 students at two universities annotated skin lesion images for hair coverage on a 3-point scale
- Activity rated more effective than traditional lectures for teaching bias; students recognized personal interpretation's role
- Main drawback: emotional unease; many students still wanted clearer guidelines, missing that disagreement is a feature
Why It Matters
This approach could transform AI education by teaching students that human judgment shapes model behavior, not just algorithms.