PixelEyes decouples perception from reasoning for precise visual search

A team of researchers from multiple institutions has introduced PixelEyes, a multi-turn visual reasoning agent that tackles a fundamental flaw in current multimodal large language models (MLLMs): their tendency to get stuck in long, redundant trajectories when trying to localize visual targets. The core insight is that existing MLLMs entangle reasoning (deciding what to look for) with perception (identifying where it is), causing inaccurate localization to trigger endless correction loops. PixelEyes explicitly decouples these two functions.

The agent introduces two key innovations. First, mask-guided visual search uses a referring segmentation model to provide mask-precise localization, freeing the reasoner from needing to compensate for imprecise grounding. Second, semantic-region breadth-first search (BFS) organizes exploration as a BFS over semantic regions, eliminating the redundant cropping of incorrect sub-regions. To internalize these capabilities, the team created the PixelEyes-6K dataset by resynthesizing expert trajectories from existing data, embedding the mask-guided search and BFS logic into the model.

The researchers also developed Pinpoint-Bench, a zero-hint visual search benchmark that provides instance-level masks and bounding boxes to separate localization failures from reasoning failures. This allows fine-grained analysis of failure modes like inattentional blindness. Current state-of-the-art MLLMs and visual reasoning agents leave large headroom on Pinpoint-Bench, confirming the challenge and value of the PixelEyes approach. The code and models are open-sourced on arXiv.