RAMP: Hybrid DRL for Online Learning of Numeric Action Models

A team of researchers including Yarin Benyamin, Argaman Mordoch, Shahaf Shperberg, and Roni Stern has introduced RAMP (Reinforcement learning, Action Model learning, and Planning), a novel hybrid strategy for learning numeric action models online. Traditional automated planning requires a predefined model of an environment's actions, which is often difficult to obtain. While models can be learned from observations, existing methods for numeric domains are offline, needing pre-recorded expert traces. RAMP breaks this limitation by learning through direct interaction with the environment, simultaneously training a Deep Reinforcement Learning (DRL) policy and refining a numeric action model from past experiences.

These components form a synergistic loop: the RL policy explores to gather data, which improves the accuracy of the learned action model. In turn, this improved model enables more effective planning, which guides further policy training. To bridge the gap between planning problems and RL environments, the team also developed Numeric PDDLGym, a framework for converting numeric planning problems into standardized Gym environments. Experimental validation on standard International Planning Competition (IPC) numeric domains showed that RAMP significantly outperforms the prominent DRL algorithm PPO (Proximal Policy Optimization), achieving higher rates of solvability and generating higher-quality plans.