TargetFinder: Real-time widget detection powers universal smart cursors

TargetFinder addresses a long-standing challenge in human-computer interaction: obtaining application-agnostic widget geometry for 'target-aware' pointing techniques. Previous approaches like Bubble Cursor or Semantic Pointing required custom APIs or accessibility hooks, limiting them to specific apps or platforms. TargetFinder uses computer vision to infer widget positions directly from screen pixels, making it truly cross-platform.

Built on a custom dataset of 520 desktop screenshots (Windows, macOS, Ubuntu, web) with nearly 38,000 annotations, TargetFinder leverages several fine-tuned YOLO models for low-latency detection. It operates at millisecond speeds, suitable for real-time interaction. In evaluations, it outperformed OmniParser and REMAUI in both accuracy and speed. The team demonstrates working implementations of Bubble Cursor and Semantic Pointing across multiple applications, proving the system's feasibility for universal deployment. The dataset, models, annotation tool, and an open-source library are publicly released.