CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

A collaborative team from Stanford University, UC Berkeley, and UT Austin has released CaP-X, a comprehensive open-source framework designed to systematically evaluate and improve AI agents that control robots through generated code. The core of the system is CaP-Gym, an interactive environment where agents synthesize and execute programs using perception and control primitives. Using their new CaP-Bench evaluation suite, the researchers tested 12 leading language and vision-language models, including GPT-4, Claude 3, and Gemini, across various levels of task abstraction.

Their key finding was a consistent trend: while AI performance improves with human-provided abstractions, it significantly degrades when these 'scaffolding' priors are removed, exposing a critical dependency. However, the team demonstrated this gap can be closed through scaled 'agentic' computation at test time. Techniques like multi-turn interaction, structured execution feedback, visual differencing, and automatic skill synthesis substantially improved robustness, even when agents operated with only low-level primitives.

These insights led to the development of CaP-Agent0, a training-free framework that recovered human-level reliability on several simulated and real-world manipulation tasks. The researchers also introduced CaP-RL, showing that reinforcement learning with verifiable rewards further improves success rates and enables effective simulation-to-real-world transfer with minimal performance loss. By providing a standardized, open platform, CaP-X aims to accelerate progress in creating more autonomous and capable embodied AI systems that can reliably code for physical robots.