Syll: Open-source agent automates APIs, CLI, and desktop GUIs

Syll, introduced by a team of researchers, is a new open-source framework for personal AI automation that breaks down the barrier between different computer interfaces. Unlike existing agents that specialize in either API calls, shell commands, or GUI clicks, Syll unifies all three in a self-hosted runtime. At its core is a bidirectional user-agent interaction layer: users can teach new procedures by simply demonstrating them on screen, and Syll compiles those demonstrations into reusable, editable skills. Conversely, every agent action is translated back into multimodal evidence—logs, keyframe screenshots, and approval checkpoints—so users can inspect, audit, and approve or override each step.

The system externalizes memory, skills, routines, and governance as plain-text and image artifacts that users can examine and modify directly. This makes Syll both transparent and extensible. The researchers validated Syll on real desktop applications including Adobe Photoshop, Adobe Audition, the game Stardew Valley, and macOS Finder, demonstrating reliable multimodal routing, teachable GUI replay, and persistent local artifact management. Syll is released as an open-source project, with the goal of providing a practical foundation for personal automation that users can teach, inspect, and continuously extend without relying on closed platforms.