CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents

A new research paper titled CUAAudit, authored by Marta Sumyk and Oleksandr Kosovan, presents a large-scale meta-evaluation of using Vision-Language Models (VLMs) as autonomous auditors for Computer-Use Agents (CUAs). CUAs are AI systems that can execute tasks on a computer desktop by following natural language instructions. The study's core problem is that evaluating these agents at scale is difficult; existing methods like static benchmarks or manual checks are brittle and costly. The researchers propose using VLMs—AI models that understand both images and text—to automatically judge whether a CUA successfully completed a task by analyzing the final state of the computer screen.

The team tested five state-of-the-art VLMs across three established CUA benchmarks spanning macOS, Windows, and Linux environments. They analyzed the models' performance along three key dimensions: accuracy in judging task success, calibration of their confidence estimates, and agreement between different models. The findings reveal a significant reliability gap: while the best-performing VLMs achieved strong accuracy and were well-calibrated in their confidence, all models showed notable performance degradation in more complex or heterogeneous computing environments. Furthermore, even high-accuracy models exhibited substantial disagreement in their judgments on the same tasks.

These results highlight fundamental limitations in current model-based auditing approaches. The inconsistency and environmental sensitivity of VLM auditors mean that deploying autonomous CUAs in real-world settings without accounting for evaluator uncertainty is risky. The study concludes that the field must develop new methods to explicitly measure and incorporate auditor reliability, variance, and confidence to build trustworthy evaluation pipelines for the next generation of autonomous AI agents.