Diagnosing LLM-based Rerankers in Cold-Start Recommender Systems: Coverage, Exposure and Practical Mitigations

A new research paper from Ekaterina Lemdiasova and Nikita Zmanovskii systematically diagnoses why LLM-based rerankers—specifically cross-encoder architectures—fail in cold-start recommendation scenarios where user history is limited. Using the Serendipity-2018 movie dataset with controlled experiments across 500 users, the study reveals three critical failure modes: extremely low retrieval coverage (recall@200 of 0.109 vs. 0.609 for baselines), severe exposure bias where rerankers concentrate recommendations on just 3 unique items versus 497 for random baselines, and minimal score discrimination between relevant and irrelevant content (mean difference = 0.098).

Surprisingly, the research shows that simple popularity-based ranking substantially outperforms sophisticated LLM reranking, with hit rate@10 metrics of 0.268 versus a dismal 0.008 for LLM approaches. The performance gap is attributed primarily to limitations in the retrieval stage rather than the reranker's capacity itself. The authors demonstrate that the problem isn't that LLMs can't understand content, but that they fail to surface diverse candidates in the first place.

Based on these findings, the paper provides practical recommendations for mitigating these issues, including hybrid retrieval strategies that combine multiple approaches, optimization of candidate pool sizes, and score calibration techniques to improve discrimination between items. All code, configurations, and experimental results have been made publicly available to ensure reproducibility and further research in this critical area of AI deployment.