LLMs hit 31% success generating specs for VeriFast; Gemini 2.5 Pro leads

Researchers from Purdue University conducted the first comprehensive study of LLM-generated specifications for separation logic (SL) verifiers, specifically VeriFast. They tested ten LLMs—including Gemini 2.5 Pro, GPT-4, and Claude—using eight different prompting strategies across three input types. The goal was to automatically produce the complex auxiliary specifications needed to verify heap-manipulating C programs, a task that normally requires significant human effort. The dataset included 303 C functions with ground-truth specifications.

The results revealed a stark gap: LLMs excelled at preserving functional behavior in both code and specifications (over 91% accuracy), but struggled with actual verification success (only 31.4%). Gemini 2.5 Pro achieved the highest verification rate, particularly when provided with formal contract inputs. The primary failure mode (94% of errors) was the LLMs' inability to correctly apply domain-specific knowledge of separation logic—such as predicates for heap structure framing. The study offers concrete guidance for improving prompt design and model training to close this gap, potentially automating a major bottleneck in industrial-scale static verification.