VistaFuzz uses LLMs to find bugs in Python libraries

A team of researchers led by Bin Duan and collaborators has developed VistaFuzz, a novel fuzzing technique that leverages locally hosted open-source large language models (LLMs) to improve the reliability of Python libraries. Unlike traditional fuzzing methods that often struggle with complex parameter constraints and inter-parameter dependencies, VistaFuzz uses LLMs to parse API documentation and generate inputs that satisfy both explicit and implicit constraints. This approach is particularly critical for Python libraries underpinning deep learning, scientific computing, and data analysis, where API reliability directly impacts downstream applications.

The researchers evaluated VistaFuzz on 7,718 APIs across twelve widely used Python libraries. They found that 40.1% of these APIs contained inter-parameter relationships that required contextual understanding to generate valid test inputs. Without resolving these dependencies, the valid generation rate for these APIs dropped from above 95% to between 31.6% and 52.8%. VistaFuzz successfully identified 74 issues, of which 43 were confirmed by library maintainers and 29 have already been fixed. This demonstrates the tool's potential to significantly enhance software reliability in the Python ecosystem.