Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation

Researchers have developed a scalable pipeline that automatically generates high-quality training data for AI web agents. Their key innovation is a constraint-based evaluation framework that provides fine-grained assessment of task progress, allowing them to use partially successful training trajectories. On a new benchmark called BookingArena—comprising complex booking tasks across 20 popular websites—their distilled student model outperforms open-source approaches and matches or exceeds commercial systems, despite being significantly smaller.