Single LLM rewrite matches manual skill tuning with 32x speedup

Enterprise AI agents route user queries to specialized skills by matching natural language descriptions. When two skills share overlapping descriptions, the routing LLM misroutes queries — a failure called skill collision. As agents scale to dozens of skills, manually tuning descriptions becomes a significant engineering bottleneck. Researchers at a major tech organization (paper on arXiv 2606.30775) deployed an automated description optimization pipeline on a production enterprise group chat agent with 9 skills and 372 regression cases. The pipeline produced descriptions averaging 79.2% F1, matching manually tuned descriptions at 79.4% F1 (average per-skill difference of -0.20%, within the 0.78% multi-seed noise floor), while reducing per-skill engineering effort from 120 minutes to 3.8 minutes — a 32x speedup.

Systematic ablation on both the production system and ToolBench (16k tools) revealed a striking result: a single LLM rewrite using any available false-positive and false-negative cases captured most of the improvement. Other design choices (iteration budget, feedback signal composition, dual editing of confused pairs, and training set size) each affected final F1 by less than 0.5%. The authors also identified a diagnostic — a large train-validation F1 gap — that flags cases where two skills' intended scopes genuinely overlap, requiring architectural intervention rather than text-level fixes. This work provides actionable guidance for engineers maintaining AI agents: invest in curating a small set of failure cases for a single rewrite, and avoid over-investing in complex multi-iteration pipelines. The 32x reduction in manual effort makes it practical to scale agent routing to hundreds of skills without sacrificing accuracy.