New R3-Skill Benchmark Improves LLM Agent Skill Retrieval with Rejection Data

LLM agents tackle complex tasks by composing skills, making skill retrieval critical. But unlike document retrieval, top-K correctness depends on whether retrieved skills work together—a property called query-conditioned skill compatibility. A new paper from researchers introduces R3-Skill, a bilingual (Chinese-English) benchmark designed specifically for agent skill routing. It contains 10,246 skills across 8 thematic super-domains, 41,592 accepted queries, and 32,828 LLM-rejected annotations organized into an 8-class rejection-reason taxonomy. Test-set ground truth was verified by multiple experts, and queries were LLM-rewritten to better approximate real user requests. The key insight: LLM rejection signals that are normally discarded as low-quality data can serve as valuable compatibility supervision.

The team also built the Reject-as-Resource Retriever (R3), a two-stage pipeline consisting of R3-Embedding (bi-encoder) and R3-Reranker (cross-encoder). Gradient analysis revealed why the rejection signal is weak in the bi-encoder objective under bilateral balancing but useful in the cross-encoder for graded ranking supervision. On R3-Skill, the full pipeline achieves Hit@1 = 0.7521, NDCG@10 = 0.8173, and a novel Set-Compat metric of 0.3188 measuring skill compatibility. The dataset, model weights, and evaluation scripts will be open-sourced.