APOLLO framework lets robots know when not to move objects, cutting LLM usage

Robotic assistants in homes typically assume clean observations and full actionability, but real-world clutter introduces ambiguity—should that coffee mug go on the shelf or stay on the desk? The new APOLLO framework tackles this by combining a lightweight personalized embedding model (PEM) that learns each user's organizational preferences from a handful of demonstrations. PEM runs entirely on CPU, making it privacy-preserving and cost-effective. Crucially, it outputs uncertainty estimates alongside placement predictions. When uncertainty is low, APOLLO acts autonomously; when high, it selectively queries an LLM for reasoning, balancing speed with judgment.

The team also created APOR, a synthetic LLM-generated dataset with room-level layouts, diverse user profiles, explicit abstention scenarios (i.e., leaving objects untouched), and noisy partial observations. Experiments show APOLLO outperforms pure LLM baselines on both PARSEC and APOR benchmarks while using up to 10x fewer LLM calls. This hybrid approach brings practical robotics one step closer to deployment in real homes—where knowing when not to act is just as important as knowing where to place things.