New Paper Exposes Hidden Failure Modes in N-Gram Generative Retrieval

Generative Retrieval (GR) replaces traditional index-based search by having a language model directly generate document identifiers. But as a new paper from Takacs, Bracher, and Vakulenko shows, this paradigm comes with its own set of failure modes that are poorly understood. The authors first build a taxonomy of GR failures drawn from existing literature, then dive deep into two prominent n-gram-based implementations: SEAL and MINDER. Their experiments reveal three recurring problems: ambiguous docids that point to multiple documents, low identifier diversity leading to repetitive outputs, and a few specific identifiers that dominate rankings disproportionately.

To help the community debug these issues, the team built an interactive web-based tool that visualizes which n-grams contribute to a given ranking and where the model goes wrong. This is especially valuable because GR systems are being deployed in production for tasks like enterprise search and question answering. The paper, currently a work in progress, is available on arXiv (2606.17721) and offers both a diagnostic framework and a practical tool — bridging the gap between research and real-world debugging.