Drift-Aware Continual Tokenization for Generative Recommendation

A research team led by Yuebo Feng has introduced DACT (Drift-Aware Continual Tokenization), a novel framework designed to solve a critical problem in modern AI-powered recommendation systems. These systems, like those used by Netflix or Amazon, typically use a two-stage pipeline: a tokenizer converts items (movies, products) into discrete identifier codes, and a generative recommender model (GRM) predicts what users want based on these codes. The challenge is 'collaborative drift'—as new items are added and user behavior patterns shift over time, the tokenizer's codes become outdated, causing recommendation quality to plummet. Fully retraining the entire AI model is prohibitively expensive, but simply fine-tuning it can break the model's existing knowledge.

DACT tackles this by balancing plasticity (adapting to new data) with stability (preserving learned knowledge). Its first stage involves fine-tuning the tokenizer alongside a newly trained Collaborative Drift Identification Module (CDIM), which intelligently detects which items have significantly drifted in user preference. The second stage uses a 'relaxed-to-strict' hierarchical code reassignment strategy to update token sequences only where necessary, minimizing disruptive changes to the majority of existing items. In experiments across three real-world datasets using two different GRMs, DACT consistently outperformed existing baseline methods. The framework's public implementation offers a practical tool for maintaining the accuracy of large-scale, live recommendation engines as they naturally evolve, providing a more sustainable alternative to constant, resource-intensive retraining.