LLMORPH: Automated Metamorphic Testing of Large Language Models

A team of researchers has introduced LLMORPH, a novel automated testing framework designed to evaluate the reliability of Large Language Models (LLMs) like GPT-4 and Llama 3. The tool addresses a core challenge in AI testing: the lack of automated 'oracles' to verify if an LLM's output is correct. LLMORPH leverages a software engineering technique called Metamorphic Testing (MT), which uses predefined 'Metamorphic Relations' (MRs) to generate new, related test inputs from an original. By checking if the model's outputs for these related inputs remain logically consistent, the system can uncover bugs without requiring expensive, human-labeled correct answers.

In a comprehensive evaluation detailed in a paper accepted to the ASE 2025 conference, the team applied LLMORPH to three state-of-the-art models: OpenAI's GPT-4, Meta's Llama 3, and NousResearch's Hermes 2. They executed a massive test suite of over 561,000 runs using 36 different MRs across four standard NLP benchmarks. The results successfully demonstrated the tool's effectiveness in automatically exposing inconsistencies and faulty behaviors that would be difficult to catch manually. The framework is designed to be extensible, allowing developers and researchers to plug in any LLM, NLP task, or set of custom test relations to rigorously assess model robustness before deployment.