Adaptive Re-Ranking cuts latency 53x by routing queries to right model

Modern information retrieval systems typically use a fixed 'retrieve-then-rerank' pipeline, applying an expensive cross-encoder reranker to all queries regardless of complexity. This wastes compute and adds latency for simple lookups. Researchers from UMass Amherst (James Allan, Ata Cinar Genc, Emir Kaan Korukluoglu) propose Adaptive Re-Ranking, a utility-based labeling framework that routes each query to different retrieval and reranking stages based on predicted cost-effectiveness. They train a routing classifier with three strategies: sparse retrieval (BM25), lightweight dense reranking (MiniLM-L6-v2), and heavy neural reranking (BGE-v2-m3). The router decides which combination to use per query, aiming to minimize latency while preserving ranking quality.

Compared to always using BGE reranking, the adaptive method achieves 1.15-53x lower median latency and 1.11-5.22x lower mean latency across all tested datasets. The nDCG@10 metric ranges from -17.5% to +4.0% relative to BGE, meaning in some datasets the adaptive approach actually improves accuracy while in others it has a modest loss. The paper provides oracle analysis showing significant potential gains from per-query routing, but notes that learning such a router from limited supervision is non-trivial. The authors demonstrate that their trained router offers a scalable, practical solution for reducing computational costs and latency in production IR systems without sacrificing user experience.