SkillResolve-Bench slashes harmful agent skill retrieval to zero
New benchmark and method eliminates risky sibling skills from agent retrievals...
Agent skill libraries are becoming routable software assets, but a retriever can find the right capability family yet expose the wrong same-capability representative — a risky sibling that leads to stale resources or wrong procedures. To measure and solve this, Jiandong Ding presents SkillResolve-Bench, an auditable benchmark with 661 helpful/risky pairs, source-role evidence, and a 7,982-candidate pool including 6,660 public SkillRet candidates. It reports harmful sibling rate (HSR@K) alongside helpful ranking.
The companion SkillResolve method resolves active candidate families, scores query-conditioned utility from confusable negatives, and selects one representative per family before the final top-K. Under the released family relation, SkillResolve achieves Recall@3 0.766 and NDCG@3 0.699 while keeping HSR@3 at 0 — a dramatic improvement over SkillRouter, which had 0.693 HSR@3. This shows that within-family representative choice is the mechanism turning capability retrieval into safer procedural exposure.
- Benchmark includes 661 helpful/risky pairs and 7,982 candidate skills
- SkillResolve achieves 0 HSR@3 versus 0.693 for SkillRouter
- Improves Recall@3 by 0.112 and NDCG@3 by 0.165 over baseline
Why It Matters
Prevents AI agents from executing dangerous or outdated skills, critical for reliable autonomous systems.