SemR-p: New metric evaluates keyphrases by meaning and ranking
A rank-aware metric that rewards semantically relevant keyphrases appearing early in output lists
Evaluating automatically generated keyphrases has long been tricky: exact-match metrics miss synonyms, while semantic similarity metrics ignore the order of predictions. In their new paper, Shamira Venturini and Steffen Kinkel propose Semantic R-Precision (SemR-p), a metric that fuses semantic similarity with the ranking awareness of R-Precision. Built from a human-centric perspective and inspired by Information Retrieval metrics, SemR-p rewards keyphrases that are both semantically relevant and appear early in the output list—matching how users actually assess usefulness.
Extensive experiments across multiple models and datasets show that SemR-p provides strong semantic sensitivity, ranking awareness, and discriminative power. It complements traditional lexical and semantic metrics, offering a more nuanced view of keyphrase quality. For practitioners in NLP and information retrieval, SemR-p means better evaluation of summarization and content extraction systems, leading to more user-aligned outputs in search, tagging, and knowledge management tools.
- Integrates semantic similarity into rank-aware R-Precision framework
- Rewards keyphrases that are both conceptually relevant and appear early in output lists
- Designed from a human-centric perspective, better reflecting user notions of relevance
Why It Matters
Improves evaluation of NLP keyphrase generation, leading to more accurate content extraction and search.