SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models

A research team led by Jianhong Li and Zeheng Qian has introduced SRBench, a groundbreaking benchmark designed to comprehensively evaluate Sequential Recommendation (SR) models powered by Large Language Models (LLMs). Published on arXiv, this work addresses critical gaps in existing evaluation methods, which often overemphasize accuracy while ignoring practical demands like fairness, stability, and efficiency. SRBench's core innovation is a unified input paradigm using prompt engineering, which levels the playing field by enabling fair comparisons between traditional Neural-Network-based SR (NN-SR) models and emerging LLM-based SR (LLM-SR) models. The benchmark also tackles the persistent challenge of extracting structured answers from unstructured LLM outputs through a novel prompt-extractor-coupled mechanism.

In their initial evaluation using SRBench, the researchers assessed 13 mainstream recommendation models and uncovered significant findings about current LLM capabilities. A key insight revealed that LLM-SR models tend to overfocus on item popularity metrics while lacking a deeper, nuanced understanding of actual item quality—a critical flaw for practical recommendation systems. By providing this multi-dimensional assessment framework, SRBench establishes a new standard for evaluating how well AI models can predict user preferences based on sequential behavior, moving beyond simple accuracy metrics to assess real-world applicability. This benchmark is poised to underpin future research by providing reliable, standardized evaluation criteria that align with both academic rigor and industry needs for deployable recommendation AI.