The Unreasonable Effectiveness of Data for Recommender Systems

A new research paper titled 'The Unreasonable Effectiveness of Data for Recommender Systems' by Youssef Abdou challenges the assumption that recommendation models hit diminishing returns with more data. The study implemented a rigorous evaluation workflow using LensKit and RecBole toolkits, testing 10 different algorithm combinations across 11 massive public datasets—each containing at least 7 million user-item interactions. Researchers trained models on sample sizes ranging from 100,000 to 100 million interactions, measuring performance using the standard NDCG@10 metric.

Contrary to expectations, the analysis revealed no observable saturation point where additional data stopped providing meaningful gains. When results were normalized for comparison, approximately 75% of the tested configurations achieved their best performance at the largest completed sample size of 100 million interactions. A late-stage slope analysis further confirmed this trend, showing consistently positive improvement rates in the final training stages across most dataset-algorithm pairs.

The findings have significant implications for how companies approach recommender system development. While much attention has focused on developing more sophisticated algorithms like neural networks or transformer-based models, this research suggests that for traditional collaborative filtering approaches, simply collecting more interaction data remains one of the most reliable ways to improve recommendation quality. The study did note weaker scaling behavior in atypical dataset cases and with the RecBole BPR algorithm, indicating that data effectiveness isn't universal but applies broadly to mainstream approaches.