Improvisational Games as a Benchmark for Social Intelligence of AI Agents: The Case of Connections

Researchers Gaurav Rajesh Parikh and Angikar Ghosal have formally introduced the popular word game 'Connections' as a novel benchmark for evaluating the social intelligence of AI agents. Published on arXiv, their paper argues that successfully playing this improvisational wordplay game requires a combination of skills that go beyond standard LLM capabilities like knowledge retrieval and deductive reasoning. The game forces agents to summarize, infer, and, most critically, gauge the understanding and cognitive states of other agents they are playing with, testing a layer of social awareness current benchmarks often miss.

The core of the benchmark involves placing AI agents, such as those powered by models like GPT-4o or Llama 3, into a constrained collaborative environment where they must communicate to solve the puzzle. The researchers show that this setup evaluates an agent's ability to demonstrate 'theory of mind'—the capacity to attribute mental states to others—a key component of human-like social intelligence. This moves AI evaluation from solitary tasks toward interactive, multi-agent scenarios that better reflect real-world applications requiring teamwork and nuanced communication.

By framing 'Connections' as a formal test, the work provides a concrete, accessible method for labs like OpenAI, Anthropic, and Google DeepMind to measure progress in building socially aware AI. It challenges the field to develop agents that don't just answer questions correctly but can also strategically collaborate, explain their reasoning to others, and adapt their communication based on inferred understanding, pushing toward more sophisticated and cooperative AI systems.