CompRank Cuts LLM Reranking Costs by 90% with Token Compression

CompRank, introduced by Xuan Lu and seven co-authors, tackles the high computational cost of using large language models as rerankers in retrieval and RAG pipelines. The key insight: ranking signals in LLMs are sparse, so most token-level computation is wasted. CompRank separates document representations from query context and candidate order, allowing document-side states to be reused across queries. It then compresses each document into a small number of segment-level tokens using a learned compressor, drastically reducing the query–document interaction cost. A CopyNet-style training objective directly ties attention-based document scoring to the reranking supervision signal, avoiding the need for a full decoding step.

Experimental results from seven BEIR datasets show CompRank retains only 10.2% of original document tokens while achieving an average NDCG@10 of 39.2, just 0.5 points below the full-token attention baseline (39.7). On TREC-COVID, the framework scales gracefully from its 30-document training regime to lists of 500 documents, with no performance degradation. The end-to-end speedup over generation-based listwise reranking reaches 4.9x–9.5x, and about 1.3x over a full-token CompRank variant. These results suggest token-level compression combined with decoding-free attention scoring is a practical path to scalable LLM reranking for large candidate lists.