Study: Popularity Calibration in Music Recs Fails to Win Users' Preference

A new arXiv paper (2608.05402) from Oleg Lesota and colleagues at TU Graz and Know-Center investigates whether popularity calibration in music recommender systems actually benefits users. The researchers recruited participants and generated personalized track lists using a controlled naive recommender to isolate popularity composition as the only variable: some lists were heavy on high-popularity tracks, some on low-popularity tracks, and some were calibrated to match each user's historical listening distribution. The study measured user preferences, perceived familiarity, and robustness of the JSD (Jensen-Shannon Divergence) metric, a standard offline measure for calibration quality.

Results reveal a split between algorithmic assumptions and real user behavior. Participants clearly perceived differences in the popularity composition of the lists—so the manipulation was successful—but they did not prefer calibrated lists over the alternatives. The authors found that the relationship between JSD and perceived popularity shifts depending on item familiarity and how much listening history is available. In fact, when familiarity or history was incomplete, the correlation weakened significantly. Additionally, comparing computational popularity labels (based on global play counts) with users' self-reported popularity judgments showed only weak alignment. This challenges the common design assumption that matching a user's past popularity profile is automatically desirable, and suggests offline metrics like JSD may not capture what users actually value.