GEB-Bench reveals AI's abstraction gap: models fail cross-voice mapping

GEB-Bench, released by Tong Zhang and colleagues at arXiv, is a new benchmark that probes a fundamental question: can AI models see the same abstract structure in a photograph of a river delta, a folk tale, and a piece of code? Named after Gödel, Escher, Bach, the benchmark builds tasks around formal motifs such as self-reference, strange loops, and Möbius twists. Each motif appears in multiple 'voices'—a natural scene, a folktale with mechanically checkable form devices, a mathematical theorem, and a programmatic skeleton. The researchers evaluate 12 open and proprietary models, examining how well they recognize structures within a single voice and map them across voices.

The central finding: recognition is strong, but cross-voice mapping consistently fails. Every model pays a 'tax' when transferring structures, and only frontier-tier models begin to narrow the gap. Error patterns align with the designed formal geometry, not perceptual geometry, and frontier models from different vendors converge on identical wrong answers. Surface complexity hurts all models, with added capacity buying headroom rather than immunity. The benchmark is fully generative, with its pipeline released, meaning it can produce endless new examples. GEB-Bench offers a way to measure abstraction as a cross-modal capability, a step beyond standard single-domain benchmarks.