MacArena benchmark exposes macOS blind spots for AI agents – top model trails by 26%

Current computer-use agents (CUAs) are often benchmarked on Linux environments like OSWorld, but macOS remains overlooked. Existing benchmarks such as macOSWorld cover only a narrow set of first-party apps and run on x86 VMs incompatible with Apple Silicon. To address this, researchers from the paper accepted at AIWILD (ICML 2026) introduce MacArena, a new benchmark comprising 421 tasks across 50 macOS applications. It reuses ported tasks from OSWorld and macOSWorld and adds 49 new macOS-native tasks, all running on Apple's native Virtualization framework on actual Apple Silicon hardware.

The key finding is striking: model rankings invert completely between the ported and macOS-native task subsets. The leading model on ported tasks trails by over 26% on the MacArena subset, indicating that strong performance on existing Linux benchmarks may reflect familiarity with task distributions rather than genuine cross-platform GUI ability. macOS introduces distinct challenges—like menu bars, Finder interactions, and system dialogs—that current agents struggle with. The paper argues that future CUAs must be evaluated in diverse, real-world OS environments to ensure robust, transferable competence.