LENS uses LLMs to declutter robotic scenes for better planning

Researchers from the University of Pennsylvania have introduced LENS (LLM-guided Environment Simplification), a plug-and-play approach that uses large language models to automatically generate task-specific scene abstractions for robotic manipulation. The problem: real-world multi-object clutter is notoriously challenging for current robotic systems due to scaling complexity, unpredictable contact physics, distractors, and task ambiguity. Traditional solutions require extensive manual engineering to create scene abstractions, which doesn't scale. LENS solves this by having an LLM dynamically merge stacked objects or prune distant, irrelevant entities in a closed loop, responding to task progress. This creates versatile, adaptively updating abstractions that can be layered on top of existing planning and control stacks.

In experiments, LENS demonstrated significant improvements across a diverse set of highly cluttered manipulation tasks. It enhanced classical planners, model-based controllers, and even modern vision-language-action (VLA) models—all without requiring retraining or manual tuning. The key insight is that LLMs can reason about which objects matter for a given task and how to simplify the environment representation, much like a human would. This makes LENS a promising bridge to real-world deployment of general-purpose robotic manipulation in homes, warehouses, and factories where clutter is the norm.