DIG framework unifies ranking and retrieval with end-to-end tokenizer training
A single training run now yields both ranking and retrieval models.
In a new arXiv paper, Shuli Wang and colleagues introduce DIG (Discrimination Is Generation), a framework that unifies ranking and retrieval by rethinking semantic IDs (SIDs). Existing tokenizers are trained independently from retrieval objectives, causing generative retrieval to lag behind discriminative ranking. DIG solves this by embedding the tokenizer directly inside a discriminative ranking model for end-to-end training. The ranker naturally becomes a retrieval model, yielding both from a single training run.
DIG organizes around a feature assignment taxonomy: item-intrinsic static features are encoded into SIDs, user-item cross features (u2i) implicitly drive codebook boundaries toward recommendation decision boundaries, and an MLP_u2t distillation module approximates u2i at the token level for inference. Experiments on three public benchmarks and two industrial datasets demonstrate that DIG simultaneously improves ranking, retrieval, and unified retrieval-ranking quality, closing the gap between generative and discriminative approaches.
- DIG embeds the tokenizer inside a discriminative ranking model for end-to-end training, unifying ranking and retrieval.
- Uses a three-part taxonomy: static item features for SIDs, user-item cross features for codebook boundaries, and MLP distillation for inference.
- Achieves simultaneous improvements on 3 public benchmarks and 2 industrial datasets.
Why It Matters
Unifies generative retrieval and discriminative ranking, enabling more efficient and accurate information retrieval systems.