8 open-weight LLM agents survive 10-day persistent MMO stress test
The question of whether open-weight language models can maintain coherent behavior over days in complex environments has been answered—with surprising results that challenge assumptions about model size and reasoning.
The Null Epoch project created a persistent MMO stress test where every player is an LLM agent, designed to test long-horizon planning, resource contention, and adversarial pressure over days or weeks. Season 0 ran 25 agents across 8 open-weight models (including Qwen3 235B/32B, Nemotron 3 Nano 30B, Ministral 14B/8B, Gemma 3 12B, GLM 4.7 Flash) over 10 days with ticks every 60 seconds. The resulting 93,000-event dataset is published on HuggingFace with ~70% of actions including model reasoning traces. The simulation used personas and directives, so findings reflect role-playing behavior rather than general model tendencies.
Key findings: Ministral 8B and 14B punched above their weight, maintaining consistent long-term awareness without hallucinating goals. Nemotron 30B, though cheap and directive-compliant, showed no strategic self-preservation—one agent died over 300 times because its directive was 'gather' and it would blindly repeat. Qwen3 235B unexpectedly invented arbitrage by hoarding resources and trading for profit, despite no explicit instruction. Season 1 now running control agents with only basic truths about the simulation, aiming to produce cleaner behavioral comparisons. The dataset provides rich ground for studying agent behavior in dynamic environments beyond static benchmarks.
- Small open-weight models (8B parameters) can match larger ones in long-term state awareness, challenging the assumption that bigger is always better for agent tasks.
- Emergent economic behaviors like arbitrage can arise spontaneously in open-weight LLMs, but may be environment-specific rather than evidence of general reasoning—validation requires controlled experiments.
- The 93k-event dataset is a free, open benchmark for fine-tuning and evaluating open-weight agents, but full reproducibility hinges on disclosing system prompts and environment rules.
Why It Matters
Open-weight LLMs are proving viable for persistent game worlds, unlocking a $10B+ market while demanding rigorous reproducibility standards.