AI Safety

AI's Labels Depend on How You Define the Problem

AI's Labels Depend on How You Define the Problem

⚡New study shows AI's answers change with the definition you give it—here's why that matters.

Deep Dive

Social scientists often use AI to analyze large amounts of text, like labeling whether a tweet is hate speech or fake news. They do this by giving the AI a definition of the concept. But many concepts have multiple valid definitions, and it's unclear if the AI actually uses the definition or just relies on its pre-trained knowledge. A new study tested this by having AI label the same texts twice with different definitions. They found that changing the definition changed the labels more than just rephrasing the same definition. This shows that AI's annotations are sensitive to the definition provided.

The study tested six concepts (including hate speech, fake news, and terrorism) and three large language models. For each concept, they swapped competing definitions and also paraphrased one definition. The swap caused more disagreement in labels than the paraphrase. This means the AI is not just using its pre-trained notion but actually applying the definition to some extent. However, the paraphrase floor (disagreement from rephrasing) shows there is some inherent randomness.

The takeaway is that any set of AI annotations should be interpreted together with the definition that produced it. Researchers should report both the definition sensitivity and the paraphrase floor to be transparent about the reliability of their labels. This is crucial for social science research that relies on AI annotation, as it affects the validity of findings. For the general public, it means that AI's judgments on complex issues are not absolute—they depend on how we define those issues, so we should be cautious about treating AI outputs as objective truth.

Key Points
  • AI's labels for concepts like hate speech change when you give it different definitions.
  • Swapping definitions caused more disagreement than just rephrasing the same definition.
  • Researchers should report which definition they used and how sensitive the AI is to it.

Why It Matters

AI is increasingly used to judge online content, but its decisions depend on how we define the problem—so we must be careful.

🔒 FREE VIP ACCESS

Read the full breakdown & get daily AI executive intelligence

Join like-minded builders, researchers, and AI professionals who read The Neural Feed daily. 100% free. No spam.

🔔 BREAKING RELEASE RADAR INSTANT ALERTS • 1-CLICK PAUSE

Get instant breaking AI release alerts by topic

Choose the AI labs, models, or chip tracks you follow. Receive fast alerts only when something major ships — plus our free daily morning briefing.

Includes full article unlock • Only major releases • Unsubscribe anytime
🎁 FREE EXECUTIVE DOWNLOAD ISSUE #21

Download This Week's 8-Page Executive AI Briefing (PDF)

Our curated executive intelligence packet: in-depth breakdowns of frontier models, market movements, plus full-color AI comics.

  • ✓ 8-page formatted executive PDF briefing
  • ✓ Frontier AI models & technical breakthroughs
  • ✓ Full-color satirical AI comic strips
Instant 1-click download • 100% free • Unsubscribe anytime
📬 Get the top 10 AI stories daily