Survey reveals how LLMs and metamorphic testing boost each other's capabilities

A new systematic survey from researchers including Zheng Zheng and Tsong Yueh Chen examines the reciprocal synergy between metamorphic testing (MT) and large language models (LLMs). The paper, published on arXiv in May 2026, reviews 93 primary studies and proposes a taxonomy with two complementary directions: MT for LLMs and LLMs for MT. LLMs introduce a severe oracle problem—their generative, probabilistic outputs make traditional test oracles impossible. MT solves this by checking necessary relations among multiple executions (e.g., if an LLM translates A→B and A→C, then B and C should preserve meaning). The survey covers MT applications for testing hallucination, fairness, robustness, code reliability, RAG systems, dialogues, and autonomous agents.

On the flip side, LLMs empower MT by automating its traditionally labor-intensive phases: metamorphic relation discovery (suggesting test relations), input transformation and synthesis (generating variants), executable test implementation (writing test code), and even agentic closed-loop testing (autonomous iterative testing). This bidirectional empowerment creates a virtuous cycle: MT makes LLMs more trustworthy, and LLMs make MT more scalable. The paper highlights future directions for building rigorous, scalable AI quality assurance methodologies, particularly for agentic systems and retrieval-augmented generation pipelines. It serves as a structured reference for researchers and engineers tackling the unique testing challenges posed by generative AI.