Emergent Formal Verification: How an Autonomous AI Ecosystem Independently Discovered SMT-Based Safety Across Six Domains
An autonomous AI system independently proposed using Z3 SMT solver for safety verification across six domains with 100% accuracy.
A research paper by Octavian Untila details how an autonomous AI ecosystem called SUBSTRATE S3 made a significant discovery. Without any explicit programming or instruction about formal methods, the system independently proposed using the Z3 Satisfiability Modulo Theories (SMT) solver to verify safety across six distinct domains. These included verifying LLM-generated code, ensuring tool API safety for AI agents, checking post-distillation reasoning correctness, validating CLI commands, verifying hardware assembly, and auditing smart contracts. This convergence happened across 8 different products over just 13 days, with low similarity (Jaccard similarity below 15%) between the AI-generated solution variants, suggesting the discovery was not a simple copy-paste but a repeated, independent finding.
From this discovery, the researchers developed a unified framework called 'substrate-guard' that applies Z3-based verification through a common API. When evaluated on 181 test cases across five of the implemented domains, the framework achieved perfect 100% classification accuracy with zero false positives and zero false negatives. Crucially, it detected real, subtle bugs that traditional empirical testing would likely miss, such as a critical INT_MIN overflow vulnerability in branchless RISC-V assembly code. The research also yielded a formal proof that unconstrained string parameters in tool APIs are inherently unverifiable. The authors argue this demonstrates that formal verification is not just a useful technique but an emergent property—a natural solution that any sufficiently complex autonomous system will converge upon when reasoning deeply about its own safety and reliability.
- The SUBSTRATE S3 AI ecosystem independently proposed using the Z3 SMT solver for safety verification across six domains (LLM code, agent APIs, etc.) without being explicitly told to, suggesting formal verification is an emergent property.
- The resulting 'substrate-guard' framework achieved 100% accuracy on 181 test cases with zero false positives/negatives and caught real bugs like an INT_MIN overflow in RISC-V assembly.
- The discovery occurred convergently across 8 products in 13 days with low solution similarity (Jaccard <15%), indicating robust, independent problem-solving by the AI system.
Why It Matters
This suggests future AI systems could autonomously develop and apply rigorous safety checks, fundamentally changing how we build reliable autonomous agents and software.