Mindstorms in Natural Language-Based Societies of Mind

A large research team, including notable AI pioneer Jürgen Schmidhuber, has formalized a framework for creating 'Natural Language-Based Societies of Mind' (NLSOM). Inspired by Marvin Minsky's seminal concept, an NLSOM is a collective of diverse, specialized AI agents—such as large language models (LLMs), vision systems, and other neural networks—that communicate and collaborate using natural language as a universal interface. This architecture overcomes the inherent limitations of single, monolithic models by enabling a 'mindstorm' of discussion and problem-solving among experts. The researchers demonstrated the power of this approach by assembling societies with up to 129 distinct members to tackle a wide array of practical AI challenges.

In their experiments, these multi-agent societies were applied to complex multimodal tasks including visual question answering, image captioning, text-to-image synthesis, 3D generation, and embodied AI. The modular nature of NLSOMs means new agents can be seamlessly integrated, paving the way for vastly larger and more capable systems. The paper, now published in the Computational Visual Media Journal, also opens a critical new research frontier by posing fundamental questions about the optimal social and economic structures for these AI societies, such as whether monarchical or democratic governance leads to better outcomes and how reinforcement learning principles can maximize collective reward.