New Robot Test Uses Fake Physics to Teach Real Safety Rules

There's a sneaky problem in AI research. When you teach a model something new, how do you know it actually learned — rather than just remembering something it saw during training? The authors solved this by inventing a world that cannot exist in any AI's memory: a 342,069-word synthetic rulebook describing its own made-up physics. This matters for jobs like pharmacy dispensing and hazardous-material handling, where the physical task is easy but the safety rules are private and getting them wrong is dangerous.

The test has two parts. First, a 500-question quiz on the invented rules. Second, a harder challenge called ORDER-SPATIAL: arranging objects in the safest order for a robot to pick up, in both familiar and totally new scenes. Out of the box, GPT-4.1 scored below random chance (a statistical score of 0.441) — its built-in assumptions actively fought the fake physics. After continued training, small models improved on both familiar and brand-new scenes, which is a strong sign of real learning rather than memorization.

Then came the robot stage. Models that passed the quiz often still couldn't produce plans a robot could execute, until they went through an extra skill-training step. After that, small, fully offline models beat GPT-4.1 — even when GPT-4.1 was allowed to look up the same rules (0.848 versus 0.606). The team demonstrated the whole loop, from seeing to acting, on a simulated industrial robot arm with a human correcting mistakes.

The lesson is blunt: how well a model does on the written test does not predict how well it plans real robot actions. The spatial task does. Practically, this points toward cheap, private, offline AI running robots in regulated workplaces — no internet connection, no leaking your company's secret safety procedures to a cloud provider, and no giant model required.