VISUALSKILL boosts Claude agents 15 points with multimodal skill library

Computer-use agents (CUAs) have reached near-human performance on standard benchmarks but still stumble on long-horizon tasks and unfamiliar software. To address this, researchers from MIT, UIUC, and Amazon have introduced VISUALSKILL, a multimodal skill library that replaces traditional text-only skill representations with a hierarchical index of visual and textual artifacts. Each skill is tailored to a target application and organized into per-topic files. Agents consume these via a load_topic MCP (Model Context Protocol) tool that fetches the relevant topic’s text and figures on demand. The construction pipeline uses two stages: authored documentation followed by live UI exploration, ensuring the skill captures actual GUI elements and workflows.

Evaluated on two CUABenchmarks, CUA-World and OSExpert-Eval, a Claude Code CLI agent backed by Claude Opus 4.6 achieved an average score of 0.456 with VISUALSKILL – a +15.3 point absolute improvement over the no-skill baseline (0.303). Critically, against a text-only control generated from the same source content, VISUALSKILL yielded a further +8.3 point gain (0.456 vs. 0.373), providing direct evidence that retaining visual figures helps agents both identify UI elements and verify workflow state after each action. This work highlights the importance of multimodal skill representations for robust agent automation and is open-sourced on GitHub.