Microsoft's Fara1.5-27B lets AI control browsers via screenshots

Microsoft Research AI Frontiers has introduced Fara1.5-27B, a multimodal computer use agent (CUA) designed to autonomously navigate and interact with web browsers. Unlike traditional automation tools that rely on the Document Object Model (DOM) or accessibility trees, Fara1.5-27B is vision-only: it takes periodic screenshots of the browser viewport and uses those images — combined with a text-based reasoning history — to decide the next action. The model outputs structured tool calls such as click (with pixel coordinates), type, scroll, visit URL, and web search, enabling end-to-end task completion. Fara1.5-27B is supervised fine-tuned from Qwen3.5-27B (27 billion parameters) using data generated by Microsoft's FaraGen1.5, a multi-agent pipeline that synthesizes web tasks, executes trajectories, and verifies outcomes before training. The model is co-designed with MagenticLite, Microsoft's lightweight agent runtime recommended for both research and production use. While the primary focus is automating repetitive web tasks — filling forms, shopping, booking travel, restaurant reservations, information seeking, and managing account workflows — Fara1.5-27B can also serve as a grounding model for other agents requiring pixel-accurate action prediction. Smaller variants (4B and 9B parameters) are also available for resource-constrained or latency-sensitive scenarios.

However, Microsoft explicitly outlines several limitations and out-of-scope uses. Fara1.5-27B is English-only (training data is exclusively English), and it should not be applied in high-stakes domains like legal, health, or financial advice where inaccurate actions could cause harm. The vision-only perception means the model can be misled by deceptive page layouts, prompt injections embedded in webpage content, or visual ambiguity in UI elements. Multi-step trajectories accumulate error — a single misclick early in a sequence can cascade into task failure. Run-to-run variance is significant, so benchmark results should be averaged over multiple attempts. The model may also hallucinate page state or misattribute information from earlier screenshots. Microsoft warns against unsandboxed deployments with access to sensitive accounts or files, and recommends thorough additional testing before any commercial or real-world production use.