Researchers Found a Big Flaw in How Netflix-Style AI Gets Tested

Researchers surveyed how recommender systems are evaluated offline — the dominant experimental paradigm in the field, which allows reproducible, cost-effective comparisons on historical interaction data. While lots of attention has gone to recommendation models and evaluation methodologies, the survey points out that the data processing decisions made before model training have received less scrutiny, even though they determine what information algorithms actually get and can affect whether results are comparable or reproducible. The survey offers a systematic, cross-domain characterisation of those practices, covering the pipeline from dataset selection and interaction representation through data preparation, multimodal feature extraction, and train-validation-test splitting, across paradigms including collaborative, sequential, session-based, graph-based, knowledge-aware, context-aware, multimodal, federated, cross-domain, contrastive-learning, and LLM-based recommendation. It also introduces a unified framework and taxonomy for describing data transformations and feature-extraction strategies, separating data preparation from extracting representations out of multimodal side information. According to the article's empirical analysis, the landscape is dominated by a narrow set of dataset-level transformations, particularly support-driven filtering, while representation-dependent transformations remain less common. The survey further identifies substantial heterogeneity in how auxiliary information is prepared and represented, plus inconsistencies in how data splitting protocols are specified, where similar labels may conceal different experimental conditions.