Ghost recommender slashes popularity bias with novel tokenization fix

Generative Recommenders (GRs) promise a unified end-to-end approach to recommendations, but they inherit a stubborn problem: popularity bias, where popular items dominate and niche content gets buried. A new study by Jun Yin et al. digs into why this happens. The team found two root causes: a token-level optimization flaw in the generative framework that amplifies popular patterns, and the undifferentiated property of item tokenization based on semantic indexes, which treats all items as equally generic tokens. This combination creates a feedback loop that favors the already-popular.

To break this cycle, the researchers developed Ghost, a generative recommender that introduces two innovations. First, asymmetric unlikelihood optimization penalizes the model for assigning high probability to popular items when they aren't relevant. Second, skeleton-founded tokenization creates structured, differentiated tokens that preserve item identity without losing the generative framework's efficiency. Extensive evaluations on three real-world datasets—using multiple SOTA baselines (no specific numbers given in abstract, but implied significant improvement)—show Ghost substantially alleviates popularity bias and promotes fairer recommendations, with only slight degradation to overall recommendation utility. The work provides both a diagnosis and a practical cure for one of recommendation systems' most persistent issues.