LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo

Researchers Ojas Jain and Dhruv Kumar have published LudoBench, a novel benchmark that uses the classic board game Ludo to stress-test the strategic decision-making of large language models (LLMs). The benchmark comprises 480 meticulously handcrafted 'spot scenarios' across 12 distinct decision categories, each designed to isolate a specific strategic choice like piece capture or safe-square navigation. To provide a gold standard for comparison, the team built a fully functional 4-player simulator that includes a principled 'Game-Theory' agent using Expectiminimax search, establishing a strategic ceiling far beyond simple greedy heuristics.

When evaluating six prominent LLMs from four different model families, the results were revealing. All models aligned with the optimal game-theory baseline only 40-46% of the time, demonstrating a significant gap in strategic reasoning. The models didn't fail randomly; they clustered into two distinct, incomplete behavioral archetypes: 'finishers' that focus on completing pieces but neglect board development, and 'builders' that do the opposite. Each archetype captured only half of a sound overall strategy.

A critical vulnerability exposed by LudoBench is LLMs' acute sensitivity to prompt framing. Researchers introduced a 'grudge framing' condition, where models were given a history of being attacked by another player. On identical board states, this narrative shift caused measurable behavioral changes, showing how easily an LLM's 'strategy' can be manipulated by context. This finding highlights a core weakness in deploying these models for autonomous, multi-step planning where consistency is key.

The complete package—including all 480 scenario prompts, the simulator code, and model outputs—is publicly available. LudoBench offers AI developers a lightweight, interpretable, and stochastic (dice-based) environment to benchmark progress in strategic reasoning, a capability essential for developing reliable AI agents for real-world applications.