AstroConcepts: A Large-Scale Multi-Label Classification Corpus for Astrophysics

A research team led by Atilla Kaan Alkan and including scientists from NASA and multiple universities has introduced AstroConcepts, a groundbreaking dataset designed to address one of AI's toughest challenges: extreme class imbalance in scientific text classification. The corpus contains 21,702 English abstracts from published astrophysics papers, each meticulously labeled with concepts from the comprehensive Unified Astronomy Thesaurus, totaling 2,367 distinct labels. What makes this dataset particularly valuable is its realistic representation of scientific terminology distribution—76% of concepts have fewer than 50 training examples, creating the severe power-law distribution that plagues real-world scientific NLP applications.

The researchers established strong baselines using traditional, neural, and vocabulary-constrained LLM methods, revealing three key insights that could reshape scientific NLP approaches. First, vocabulary-constrained LLMs achieved competitive performance against domain-adapted models, suggesting parameter-efficient methods might be sufficient for specialized domains. Second, while domain adaptation showed relatively larger improvements for rare terminology, absolute performance remained limited across all methods, highlighting the fundamental difficulty of the problem. Third, the team proposed frequency-stratified evaluation—a method that reveals performance patterns hidden by aggregate scores—making robustness assessment central to scientific multi-label evaluation.

By releasing this resource publicly, the team enables systematic study of extreme class imbalance and establishes concrete benchmarks for future research. The findings offer actionable insights for researchers working on scientific document classification, knowledge organization, and specialized domain adaptation, potentially influencing how AI systems are developed for technical and scientific literature analysis across multiple disciplines.