Stance Labels Fail When They Matter Most: The Projection Problem in Stance Detection

A new research paper by Bowen Zhang reveals a fundamental flaw in how AI models are trained to detect human stances on social media. The study identifies what Zhang calls the 'projection problem'—when annotators must compress multi-dimensional opinions into simple 'Favor, Against, or Neutral' labels, they inevitably weight different dimensions differently. For example, someone might accept climate science (support on science dimension) while opposing carbon taxes (opposition on policy dimension). Different annotators might label the same text differently based on which dimension they prioritize, creating training data that's inherently contradictory where it matters most.

Zhang's analysis of the SemEval-2016 Task 6 dataset shows this problem has measurable consequences. On texts where all dimensions align (consistent opinions), label agreement reaches Krippendorff's α=0.307. However, on dimension-conflicting texts, agreement collapses to α=0.085—essentially random. Meanwhile, agreement on individual dimensions remains intact, with policy dimension agreement reaching α=0.572. This reveals that the problem isn't annotator confusion but a fundamental mismatch between how humans express complex opinions and how AI systems are trained to categorize them.

The implications are significant for any application relying on stance detection, from political analysis to brand monitoring. Current models trained on these flawed datasets will perform worst precisely on the controversial, nuanced topics where accurate understanding is most critical—climate change debates, political polarization, or vaccine discourse. The research suggests that moving to multi-dimensional labeling or developing new approaches that capture opinion complexity could dramatically improve AI's ability to understand human discourse in messy real-world contexts.