Monty boosts LLM autoformalization precision by 20 points
New framework Monty turns natural language assertions into precise executable code.
Formal contracts are critical for software testing and verification, but writing them manually is time-consuming and error-prone. A new paper from Hongyi Liu, Madhusudan Parthasarathy, and Adithya Murali tackles this with Monty, an autoformalization framework that leverages LLMs to bridge the gap between informal developer intent and formal executable specifications. Monty addresses two core challenges: ensuring the validity of generated assertions and resolving ambiguity in natural language. It introduces a conformance score metric that measures how well an LLM-generated formalization aligns with the natural-language intent, and then filters candidates by actually testing the code against those formalizations to compute validity scores.
The researchers evaluated Monty on 541 assertion-generation tasks derived from 22 collection-like Java classes. Compared to using LLMs naively to translate assertions, Monty produced the ground truth more reliably, improving precision by an average of 20 percentage points. This work demonstrates that combining LLM outputs with structured validation can dramatically improve the reliability of autoformalization. For professionals, Monty means less manual contract writing and fewer bugs from incorrect or ambiguous specifications—a practical step toward fully automated software verification.
- Monty uses a conformance score metric to measure how well LLM-generated assertions match natural language intent, then validates them via testing.
- Tested on 541 assertion tasks from 22 Java collection classes, improving precision by up to 20 points over naive LLM translation.
- Code, data, and resources are publicly available to encourage further research and adoption.
Why It Matters
Autoformalization with reliable validation reduces manual contract writing and catches ambiguous specs earlier.