SkillConsist catches hidden dangers in LLM agent skills with 87.9% F1

Agent skills are reusable capabilities that let LLM agents perform tasks, but they can hide dangerous behavior or cause wrong skill selection when their declared purpose doesn't match their actual implementation. Existing consistency detection methods often evaluate behavior against predefined categories or scopes, but they struggle when declarations are concise and implementations span multiple connected steps. SkillConsist, a new method from Chaofan Meng and co-authors, tackles this by separating declaration and implementation content into behavior records using an LLM, supplemented by static analysis, then building two separate behavior graphs.

The core innovation is bidirectional graph alignment. Starting from any behavior record on either the declaration or implementation side, SkillConsist searches the opposite graph for a candidate subgraph and expands it along behavior relations until the source behavior is fully expressed. Graph differencing then identifies conflicts between the aligned subgraphs and outputs precise detection results. This approach bridges the gap between high-level descriptions and low-level code, catching inconsistencies that text-only or scope-based methods miss.

To evaluate, the team built a benchmark of 633 skills from ClawHub, including 500 most-downloaded public skills and 133 Skill-Inject packages. The dataset includes 319 inconsistent and 314 consistent skills, plus 442 localized inconsistency annotations. SkillConsist achieved 86.85% precision, 89.03% recall, and 87.93% F1 for package-level detection — a 20.43 percentage point improvement over the best baseline. For localization, it reached 67.60% precision, 58.14% recall, and 62.52% F1, showing useful granularity in pinpointing exact problem spots.

This work addresses a growing safety concern as agentic AI becomes more widely deployed. By automatically verifying that an agent skill's declared behavior matches its real-world implementation, SkillConsist could become a standard part of the pipeline for auditing and approving reusable skills — before they're allowed to act on behalf of users.