Token-level rewards in RL improve generative document retrieval

Generative retrieval models work by autoregressively generating document identifiers (DocIDs), where each decoding step selects a token and the full sequence determines the retrieved document. But retrieval effectiveness is only judged after the entire DocID is generated, creating a mismatch: token-level generation decisions receive document-level feedback. Existing reinforcement learning methods propagate the same sequence-level reward to every decoding step, making it hard to identify which token choices actually caused success or failure.

To fix this, the researchers introduce a fine-grained RL framework that estimates token-level relevance rewards. They measure how each individual token decision changes the expected retrieval quality of the generation trajectory, then use those step-wise rewards in policy optimization. This gives the model clearer guidance to favor tokens that contribute directly to document-level relevance. Experiments on retrieval benchmarks show the method consistently beats sequence-level reward baselines, proving that fine-grained supervision better aligns DocID generation with retrieval objectives—a practical step toward more accurate and explainable generative search systems.