CORE-Bench tests coding agents on repo-level code retrieval

A team of researchers from China has introduced CORE-Bench, a comprehensive benchmark designed to evaluate code retrieval abilities in the context of agentic coding agents. Unlike existing benchmarks that focus on docstring-to-function or snippet-level matching, CORE-Bench tests an agent's ability to navigate a full repository state, locate relevant files and functions, gather supporting context, and filter out similar in-repository distractors. The benchmark covers three key levels of retrieval: code understanding, issue-to-edit localization, and broader context retrieval.

CORE-Bench contains over 180,000 queries and 106,000 broader-context relevance labels, built from curated code-search tasks and SWE-bench-series instances. Experiments with representative embedding models reveal a sharp performance drop when moving from traditional code search to the more realistic agentic coding setting. However, simple supervised fine-tuning of existing embedding models significantly improves performance, indicating substantial room for progress. The paper is available on arXiv and provides a valuable resource for advancing code retrieval in agentic systems.