NASA researchers launch scientific code search benchmark with 5,264 curated repos

A team of researchers led by Nishan Pantha, including experts from NASA and academic institutions, has introduced a curated corpus and benchmark for scientific code search. The dataset comprises 5,264 high-quality, domain-classified scientific repositories spanning all five NASA Science Mission Directorate divisions: Earth Science, Astrophysics, Planetary Science, Heliophysics, and Biological & Physical Sciences. Each repository comes with cleaned READMEs, extracted topics, and additional context from crawled links, addressing the challenge of discovering relevant software among over 600 million GitHub repositories.

To evaluate search performance, the team built two novel information retrieval benchmarks. The first is a repository search benchmark with 219 expert-curated queries designed by domain scientists, exposing significant performance variation across scientific domains. The second is a large-scale code snippet retrieval benchmark containing 117,950 code snippets and 119,720 queries across seven programming languages. Baseline evaluations show that snippet retrieval is particularly challenging due to differing documentation practices, coding standards, and language conventions across scientific communities. All datasets and benchmarks are publicly released on HuggingFace to accelerate research on scientific tool discovery.