Study finds LLMs in recommendation rely on memorization, not generalization

A new arXiv study by Sunwoo Kim and colleagues reveals that large language models (LLMs) fine-tuned for generative recommendation (GR) primarily rely on memorizing one-hop item transitions from training data, rather than leveraging their rich pretrained knowledge for generalization. The authors show that LLMs exhibit this one-hop memorization behavior more than traditional non-LLM GR models. In fact, the vast majority of LLMs' performance gains over baseline GR models come from users whose target items can be predicted through simple one-hop memorization—suggesting LLMs are not using their broader knowledge as expected.

To address this limitation, the team proposes IIRG (Implicit Item Relation Graph), a novel training strategy that teaches LLMs to capture two complementary types of item relations: collaborative relations derived from item co-occurrences across multiple hops in user sequences, and semantic relations among items with similar themes. IIRG significantly improves over standard next-item prediction training, delivering especially large gains for users whose test items are not covered by training-time one-hop transitions. This work challenges the assumption that LLMs inherently bring generalization to recommender systems and highlights the need for targeted training strategies to unlock their full potential.