ValueFormer lifts robot sandwich task success from 70% to 85%

Vision-Language-Action (VLA) policies trained via behavior cloning often fail silently—a collapsing rollout looks identical to one making progress in the action stream. That's because imitation learning provides no notion of progress. Reinforcement learning could help, but real-robot experience is costly and deformable food (like sandwich ingredients) resists simulation. The paper argues the hard part isn't the architecture—it's the per-frame label: it must be dense, continuous, and correctly shaped.

ValueFormer solves this with a compact, policy-agnostic causal transformer over a frozen DINOv3 backbone. In a single forward pass, it emits two complementary signals: a smooth Monte Carlo value (V_mc) for advantage estimation, and a sharp binary value for online mistake detection. Failed episodes are labeled with a stage-aware, success-then-decay return that preserves the success curve before the failure stage, and detection is supervised from mistake intervals rather than a single failure time—so even mistakes the policy recovers from carry signal. On a real-robot bimanual sandwich-assembly task (1,427 episodes), a critic-derived per-frame training weight lifted task completion from 70% to 85% (within noise at n=20). The batched bf16 encoder cuts live serving cost 3–5x, letting the critic run at 2 Hz alongside the policy on a single GPU.