Study reveals LLM personalization fails with real human data

A new paper from researchers Lechen Zhang, Jiarui Liu, and Tal August, titled "Re-Centering Humans in LLM Personalization," uncovers a troubling gap in how large language models handle personalization. While most existing evaluations rely on synthetic data that shows strong performance, this study collected real human conversations (550 total) and over 19,000 human judgments across three critical stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and generating personalized responses (1,101).

At every stage, models fell short when evaluated by humans. In the extraction phase, models struggled to accurately pull user attributes from natural conversation. For relevance pairing, model and human judgments on which attributes to use for a given prompt often diverged significantly. Most strikingly, in the generation stage, humans rated the LLM's personalized responses as no better than generic responses — yet the models' own reward systems rated the personalized versions as vastly superior. This suggests a fundamental misalignment between what models consider effective personalization and what humans actually find useful.

The authors introduced two lightweight training-based interventions that improved alignment with human judgments in the first two stages, demonstrating it's possible to shift automated evaluation closer to human preferences. However, for the generation stage, even learned reward models achieved only modest correlation with human ratings, indicating that modeling human-aligned personalization quality is a much harder problem. The paper provides a rich dataset and framework for future research on extracting, selecting, and incorporating user information in ways that real humans find valuable.