Whose Values? Measuring the (Subjective) Expression of Basic Human Values in Social Media
Researchers tackle AI's biggest blind spot: measuring subjective human values like humility and equality in social content.
A research team from Stanford University and MIT has published a novel framework for measuring the subjective expression of basic human values in social media content. The work, titled "Whose Values? Measuring the (Subjective) Expression of Basic Human Values in Social Media," addresses a core challenge in AI alignment: how to quantify value-laden content when perceptions vary widely between individuals. The researchers grounded their work in the established Schwartz value system, which includes values like humility, equality, and power, and built a massive dataset of 32,370 human annotations on 5,211 real social media posts from 1,079 participants.
They discovered a critical problem: inter-rater agreement among humans was low, and agreement between human raters and standard LLM-based classification methods was even lower. This confirmed that value expression is inherently subjective. In response, the team developed a personalized AI architecture. Instead of seeking a single "correct" label, their model is calibrated with a small set of annotations from a specific user, learning that individual's unique perspective on what constitutes, for example, a post about "tradition" or "benevolence." In evaluations, this personalized approach succeeded—its predictions aligned with individual users' judgments better than the judgments of other people did.
- Built a massive dataset of 32,370 value annotations from 1,079 people on 5,211 real social media posts.
- Found low agreement between humans and LLMs, proving value detection is highly subjective.
- Created a personalization architecture that, after minimal user calibration, predicts values an individual agrees with more than other people's judgments.
Why It Matters
Enables more nuanced, user-aligned content moderation and recommendation systems, moving beyond one-size-fits-all AI judgments of sensitive content.