Object Search in Partially-Known Environments via LLM-informed Model-based Planning and Prompt Selection

A team from the University of Texas at Arlington has developed a new AI framework that significantly improves a robot's ability to find objects in environments it doesn't fully know. The core innovation is a hybrid approach called "LLM-informed model-based planning." Instead of just asking a large language model (LLM) where to look, the system uses the LLM to generate statistical likelihoods of an object's location. These probabilities are then combined with practical travel costs extracted from the environment's map to create an optimal, cost-aware search plan. This method effectively uses the LLM's common-sense reasoning about object placement (e.g., a mug is likely in a kitchen) while grounding it in the physical realities of movement and distance.

The paper also introduces a novel prompt and LLM selection method that works during deployment. Using a "bandit-like" selection approach based on offline replay data, the system can quickly test and choose the best prompt or even the best underlying LLM for the specific search task at hand. This adaptive component led to a 6.5% lower average search cost and a 33.8% lower cumulative regret compared to a standard bandit selection baseline. The framework was validated in both simulation and real-robot experiments in an apartment setting, demonstrating robust performance improvements over strategies that rely solely on LLMs or simple optimistic exploration.