8 open-weight LLM agents survive 10-day persistent MMO stress test

Eight open-weight large language models (LLMs) were dropped into a persistent massively multiplayer online (MMO) game for ten days, generating a 93,000-event dataset that is now publicly available. The Null Epoch experiment, run by FirespawnStudios, included models ranging from Qwen3 235B to Ministral 8B and Nemotron 30B. Key findings upend conventional wisdom: Ministral 8B showed unexpected long-term state awareness despite its small size, while Nemotron died over 300 times due to reckless behavior. Most strikingly, Qwen3 235B spontaneously developed an arbitrage strategy, buying low and selling high in the game's economy. Season 1 of the experiment now includes control agents for cleaner comparisons, establishing a new baseline for evaluating open-weight LLMs in long-horizon environments.

This experiment fills a gap left by earlier benchmarks. AgentBench from Berkeley and Tsinghua tests models on short, isolated tasks across games and websites. NVIDIA's Voyager uses GPT-4 in Minecraft but in a single-player setting with a closed-source model. Commercial platforms like Inworld AI create scripted NPC behavior with proprietary models. Null Epoch differs fundamentally: it stresses multiple open-weight models in the same persistent, multi-user world for days, emphasizing long-term memory, survival, and emergent behavior. The release of the full event log on HuggingFace mirrors the open-benchmark trend seen with LMSYS Chatbot Arena, allowing the community to audit and build upon the results.

The implications are significant but nuanced. The strong performance of Ministral 8B suggests that smaller open-weight models, with proper context management, can handle complex game dynamics—potentially slashing costs for game developers eyeing the $10 billion NPC market by 2030. However, the experiment's hidden risks temper the enthusiasm. The long-term state awareness could partly stem from engineered system prompts that are not disclosed. Only eight models were tested, many from the same architecture families (e.g., two Ministral variants). The arbitrage behavior may be an artifact of the game's toy economy rather than a sign of genuine strategic reasoning. And Nemotron's high death toll reminds us that larger models do not guarantee survival instincts; reward shaping and environment design remain critical. This stress test sets a new standard, but reproducibility requires full disclosure of prompts, memory management, and environment rules.

The bottom line: Open-weight LLMs can survive and even thrive in persistent game worlds, but the devil is in the details of context engineering and environment design. The Null Epoch dataset is a valuable resource for fine-tuning and evaluating agents, but it also underscores that observed capabilities may not transfer to other settings. As Season 1 continues with control agents, the community will gain deeper insights into what open-weight models can truly achieve—and where they still fall short. For developers and researchers, the lesson is clear: long-duration stress tests are the new frontier for agent evaluation, and open-weight models are ready to compete.