Study shows personalization works for document selection, not sentence salience

A new study from researchers Kazuki Nakayashiki and Keisuke Watanabe, published on arXiv, rigorously tests the limits of personalization in social web highlighting. Using a co-readership identity control — where the same document is highlighted by many users — the authors isolate whether a person's own reading history predicts their highlights better than another reader's. At the document level, personalization yields a clear win: an own-versus-other gap of +0.169 against community negatives and +0.119 against topic-matched hard negatives. The signal is largely thematic, not title-driven.

However, at the sentence level, the story flips. The researchers implemented a two-stage personalized auto-highlight system: an impersonal model proposes candidate sentences, then a personal model re-ranks them. This approach consistently failed to beat an impersonal baseline. Even frontier LLMs (zero-shot) could not predict highlight locations better than a simple lead baseline (first sentences). Personal re-ranking was outperformed by salience order alone, even on the most generous candidate pool. The authors conclude that measurable personalization is modest (~+0.13), topic-dominated, and appears only at the selection layer — not the salience layer. They suggest that improving shared salience may be more effective than trying to over-personalize.