Multi-level meta-reinforcement learning with skill-based curriculum

Researchers Sichen Yang and Mauro Maggioni from Johns Hopkins University have introduced a novel framework for multi-level meta-reinforcement learning that systematically breaks down complex sequential decision-making problems. Their method works by repeatedly compressing Markov Decision Processes (MDPs), where learned policies at one level become single, reusable actions in higher-level MDPs. This hierarchical compression preserves the semantic structure of the original problem while dramatically reducing unnecessary stochasticity and shrinking the policy search space, leading to computational efficiency gains of 10-100x in some cases.

The framework operates within a curriculum learning paradigm where a "teacher" agent organizes tasks by gradually increasing difficulty. A key innovation is the factorization of policies into problem-specific embeddings and transferable skills, including higher-order functions. This enables skills learned in one context to be applied across different problems and levels, creating new transfer opportunities. The researchers demonstrated these capabilities in MazeBase+, a complex navigation environment, showing how their approach enables AI agents to master intricate tasks through systematic abstraction and skill reuse.

The mathematical foundation guarantees consistency under mild assumptions, ensuring the compressed higher-level MDPs remain solvable with existing algorithms. By decoupling sub-tasks and coarsening spatial or temporal scales at higher levels, the method makes it feasible to find long-term optimal policies for problems that would otherwise be computationally intractable. This represents a significant advance toward creating AI systems that can efficiently learn and transfer complex skills across domains.