Qwen3.6-27B matches larger models in fantasy RPG agentic benchmark
Qwen's 27B model scores 82% vs Gemma-4-31B's 87%, but sub-scores reveal hidden cliffs.
A Reddit user (u/UsedMorning9886) built and ran an agentic benchmark for medieval fantasy RPG scenarios across eight local models. The suite tested quest completion, scene endings, item/time tracking, character detection, storytelling, and drafting, scored by an external LLM grader with varying sample sizes per category. Top-line pass rates: Gemma-4-31B led at 87%, closely followed by Qwen3.6-27B at 82%, and Gemma-4-12B at 80%. Smaller models (55–70%) fell off steeply.
More revealing were the sub-scores: some models that excelled at completing quests collapsed on tasks like 'NPC thoughts' or 'quest summarization'. This unevenness is invisible in overall averages. The benchmark highlights that for agentic task design—especially in interactive fiction or game AI—aggregate scores can mislead. Developers should evaluate models across specific skills rather than relying on a single pass rate. Qwen3.6-27B's strong showing suggests that efficient architectures can punch above their weight in complex role-playing scenarios.
- Qwen3.6-27B achieved 82% pass rate, just 5 points behind the 87% of much larger Gemma-4-31B.
- Sub-category analysis revealed stark cliffs: models strong on quest completion often failed at NPC reasoning or summarization.
- Benchmark covered six agentic tasks (quests, scenes, items, characters, storytelling, drafting) with varying N per category.
Why It Matters
For developers building AI-driven RPGs, this shows model size isn't everything—category-level testing is critical.