AgentOdyssey tests AI agents with open-ended long-horizon text games
Even the best AI agent is far behind humans in this new benchmark
AgentOdyssey is a new evaluation framework from researchers at Johns Hopkins University and collaborators that procedurally generates open-ended text games for testing test-time continual learning agents. Unlike traditional benchmarks that separate training and inference, AgentOdyssey forces agents to learn continuously during deployment—exploring, acquiring new knowledge and skills, retaining episodic experiences, and planning over long horizons. The games feature rich entities, dynamic world interactions, and tasks that require hundreds of steps to complete, pushing agents beyond simple pattern matching.
The framework goes beyond game progress metrics with a multifaceted evaluation: it measures world knowledge acquisition (how well the agent understands game rules and entities), episodic memory (recalling past interactions), exploration diversity (novel objects/actions tried), and computational cost. The researchers tested several agent paradigms, including language model-based agents with varying memory and reasoning mechanisms. Results show that performance scales with stronger base models (e.g., GPT-4-level), but even the best agent is still far below human performance. A critical finding: short-term memory modules consistently improved all agent types, making it a key component for test-time continual learning.
- AgentOdyssey procedurally generates unlimited open-ended text games with rich entities and long-horizon tasks (hundreds of steps).
- Evaluates agents on world knowledge acquisition, episodic memory, exploration diversity, action diversity, and model cost.
- Short-term memory significantly boosts performance across agent paradigms, yet even top agents remain far below human baselines.
Why It Matters
This benchmark exposes critical gaps in AI's continual learning, guiding future work on more adaptive, real-world agents.