Beyond Retrieval: A Multitask Benchmark and Model for Code Search

Code search in production environments is far more than a simple first-stage retrieval task—it involves reranking, handling developer-style queries, and dealing with ambiguous or short queries. Existing benchmarks often suffer from data contamination, label noise, and binary relevance judgments that don't reflect real-world needs. A new paper by Siqiao Xue et al. introduces CoREB (Code Retrieval and Reranking Benchmark), built from counterfactually rewritten LiveCodeBench problems across five programming languages. The benchmark uses timed releases with graded relevance judgments to simulate a more realistic code search pipeline, covering three tasks: text-to-code, code-to-text, and code-to-code. This multitask design helps evaluate the full spectrum of code search capabilities.

The study benchmarked eleven embedding models and five rerankers, revealing striking findings. Code-specialized embeddings dominate code-to-code retrieval by roughly 2x over general-purpose encoders, yet no single model wins across all three tasks. More critically, short keyword queries—the most common developer search format—cause every evaluated model to achieve near-zero nDCG@10, exposing a major weakness in current approaches. Off-the-shelf rerankers exhibit severe task asymmetry, with a 12-point performance swing between code-to-code and other tasks. The authors' fine-tuned CoREB-Reranker is the first to achieve consistent positive gains across all three tasks, addressing the previously unmet need for a universal code search reranker. This work provides both a more accurate benchmark for the community and a practical model that improves code search in real-world scenarios.