GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers

A team of researchers has introduced GBQA (Game Benchmark for Quality Assurance), a new benchmark designed to rigorously evaluate whether large language models (LLMs) can function as autonomous Quality Assurance engineers. The benchmark, accepted at ICLR 2026, contains 30 games with 124 human-verified bugs across three difficulty levels, created using a scalable multi-agent system. It provides a standardized testbed to measure an AI's ability to discover software bugs in complex, dynamic runtime environments—a task far more challenging than static code generation.

To run the benchmark, the researchers also developed a baseline interactive agent equipped with a multi-round ReAct loop and memory, enabling long-horizon exploration of game environments for bug detection. Extensive testing on frontier LLMs revealed that autonomous bug discovery is still a formidable problem. The top performer, Anthropic's Claude 4.6 Opus operating in a special 'thinking' mode, successfully identified less than half of the verified bugs, achieving a score of just 48.39%.

The results underscore a critical gap in current AI capabilities for autonomous software engineering. While LLMs excel at generating code from specifications, dynamically testing that code in a live environment to find runtime errors and logical flaws presents a much higher-order challenge. The GBQA benchmark establishes a concrete performance target and evaluation criterion, pushing the field toward developing AI agents that can not only write software but also reliably ensure its quality.