LM2Alloy uses LLMs to generate formal specs, catches missed bug in Flipper

A new study from researchers Tasmim Rashid and Muhammad Zubair Malik explores using Large Language Models (LLMs) to create formal specifications in Alloy—a modeling language for software design—from both natural-language requirements and production source code. Their pipeline, dubbed LM2Alloy, then automatically derives executable test cases from those formal specs. They evaluated the approach on two real open-source Python libraries: Flipper (a feature flag management system) and Cerberus (a data validation library). In both cases, the LLM generated working Alloy specifications and tests without any manual correction.

For Flipper, the pipeline uncovered a genuine bug that the existing test suite had missed: the library silently accepts duplicate flag names, directly contradicting its documented uniqueness requirement. A direct LLM baseline—generating tests from the same README but skipping the Alloy step—achieved only 68% branch coverage and failed to catch this bug across all three independent runs. This suggests that introducing a formal intermediate representation can surface constraint-level defects that coverage-oriented generation may overlook. For Cerberus, the code-derived specification captured an implicit abstraction over sized types that the documentation-derived spec omitted, producing two additional tests. Across both libraries, code-based specifications showed lower variance in test generation (mean SD = 2.15) than documentation-based ones (mean SD = 5.0), though the authors caution that whether this generalizes remains an open question.