Measuring Understanding Through Discrete Compositional Knowledge Structures in Hierarchical Automata

How do we know if an AI truly understands something versus just predicting the right answer? Igor Balaz's new paper, accepted at AGI 2026, tackles this measurement gap head-on. Current approaches—probabilistic systems, practice-based compilation, and neural embeddings—all lack inspectable structural traces of understanding. Balaz proposes building hierarchical automata from finite state machines that represent patterns, with higher-order automata encoding compositions. Constrained inference constructs these automata from single observations, while similarity detection clusters related structures to quantify concept robustness. Graph memory makes compositional knowledge directly inspectable, and metacognitive mechanisms enable observable reconfiguration. The result: understanding becomes a measurable property with distinct signatures that go beyond statistical correlation.

Balaz demonstrates the framework on a simple geometric domain, tracking graph evolution to reveal five measurable signatures: immediate representation formation, structural knowledge (how concepts are organized), generalization capacity (ability to apply to new instances), compositional awareness (understanding of parts and wholes), and metacognitive access (awareness of one's own knowledge). These signatures allow researchers to distinguish when a system genuinely grasps a concept versus when it merely matches patterns in training data. For AI practitioners, this opens the door to building systems with verifiable comprehension—critical for safety-critical applications like medical diagnosis, autonomous driving, or scientific reasoning. While still early, Balaz's work complements neural perception and neurosymbolic execution by adding a rigorous, mathematical way to audit what an AI actually understands.