AEVAL framework brings deterministic CI testing to agentic AI skill workflows

A new paper on arXiv (2607.16345) presents AEVAL (Agentic Evaluation), addressing a critical pain point in the growing ecosystem of agentic AI skills—installable packages of natural language and code that teach LLM agents domain tasks. Traditionally, evaluation has been anecdotal: a developer runs a demo, watches the agent, and forms a subjective judgment. This approach lacks reproducibility, breaks comparability across versions, and scales poorly for marketplaces where a single regression can silently break dozens of downstream workflows.

AEVAL replaces this with a deterministic, CI-integrated test pipeline. Every skill change triggers a test event where the skill runs against a developer-declared evaluation contract inside an automated executor, producing a structured, evidence-grounded quality signal. A key innovation is the structural separation between executor and grader, which prevents the subtle failure mode where an agent silently self-corrects during execution and then grades its own patched outputs as passing. The framework also introduces a tiered, grounded-evidence fix-suggestion scheme (LV1 causal, LV2 quality) posted as inline merge-request comments. Validated on real skills in a production agentic stack across multiple agent SDKs, AEVAL converts spurious 100% pass rates into reproducible first-attempt fail signals with an auditable record of every executor fix.