CIKM study shows anchor-based LLM rerankers only help with BM25 retrievers

Anchor-based pointwise LLM reranking scores each candidate against a shared reference passage to recover cross-document context at pointwise cost. In a reproduction-first study, Ghosh and Chatterjee found that the original GCCP/PAGC paper's reported 0.66 nDCG@10 was not achievable from text alone — their reimplementation got only 0.24. They identified eight undocumented implementation details that, once recovered, allowed them to reproduce results within 1.6%. This highlights how fragile LLM reranking results are to hidden design variations.

The authors then ran a controlled component-level stress test. After rigorous statistical correction, the core contrastive scoring idea held up. However, combining the contrastive score with a standard pointwise relevance score only helped when the first-stage retriever was BM25. With a stronger dense retriever like E5, this combination provided little or no benefit. This boundary condition was not visible in the original evaluation, which used weaker retrievers.

They also tested anchor construction methods. The paper's complex method for building the anchor passage was unnecessary. A much simpler anchor, created by interleaving the top-ranked sentences, matched or outperformed the original across datasets. These findings stayed consistent across LLM backbones, including a 4-bit quantized 72B model, suggesting the results are not specific to one model.

Overall, anchor-based pointwise reranking is effective, but its gains stem primarily from contrastive scoring rather than complex aggregation and anchor construction. The practical takeaway: if you are using a dense retriever, don't expect this technique to add much; if you are on BM25, use a simple anchor.