New study finds vanilla web agents beat token-hungry skill modules

A new arXiv paper challenges the assumption that adding skill and memory modules to online web agents is always beneficial. The researchers compared three popular augmentation techniques — AWM, ASI, and ReasoningBank — against a simple token-matched vanilla baseline across multiple benchmarks (WebArena and WorkArena-L1) and three models (Gemini 3 Flash, GPT-5.4-mini, and Qwen 3.6-27B). Under a fixed total inference budget, the vanilla baseline matched or surpassed all augmented methods in aggregate success rate, often while actually using fewer tokens. The authors note that these augmentation modules consume test-time tokens that are rarely accounted for in typical evaluations, and that their apparent gains vanish when the budget is controlled.

The study also reveals that run-to-run variance materially affects outcomes, suggesting that reporting average performance alone can be misleading. For enterprise knowledge-work tasks (WorkArena-L1) tested with Qwen 3.6-27B, the trend persisted: extra modules didn't deliver a net benefit. The authors conclude that while skills and workflow memory can help in specific domains, their value should be reassessed under realistic, budget-constrained conditions. This work has immediate implications for anyone building cost-sensitive web automation agents — the simplest approach might be the most efficient.