GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification

A research team led by Iordanis Fostiropoulos has introduced GISTBench, a novel benchmark designed to evaluate Large Language Models' (LLMs) ability to understand users based on their interaction histories, a critical task for AI-powered recommendation systems. Unlike traditional benchmarks that measure simple prediction accuracy, GISTBench focuses on evidence-based interest verification. It proposes two new metric families: Interest Groundedness (IG), which decomposes into precision and recall to penalize hallucinated interests and reward coverage, and Interest Specificity (IS), which assesses the distinctiveness of user profiles generated by the LLM. The team released a synthetic dataset built on real user interactions from a global short-form video platform, containing both implicit and explicit engagement signals with rich textual descriptions.

The researchers validated their dataset against real user surveys and then evaluated eight open-weight LLMs spanning from 7B to 120B parameters. The findings reveal significant performance bottlenecks in current models. A key weakness identified is the models' limited ability to accurately count and attribute engagement signals across heterogeneous interaction types, leading to unreliable user interest profiles. This benchmark shifts the focus from whether an LLM can recommend an item to whether it can correctly infer and verify *why* a user might be interested, based on concrete evidence from their history. The work highlights a fundamental challenge in deploying LLMs for personalized systems where understanding nuanced user intent is paramount.