ARMeta uses multi-agent LLMs to automate metamorphic testing for REST APIs

Testing REST APIs is notoriously difficult because verifying output correctness—known as the oracle problem—is often ambiguous. Metamorphic testing tackles this by defining relations between outputs across multiple test executions rather than checking single outputs. Now, researchers have built ARMeta, a tool that automates this process using a multi-agent LLM workflow. The agents collaborate to identify metamorphic test scenarios from OpenAPI specs, write them in structured Given-When-Then format, and generate executable tests automatically.

ARMeta was evaluated on two publicly available web applications and compared against a scenario-based testing baseline. The results show that ARMeta explores complementary behaviors not covered by traditional scenario-based testing, effectively expanding test coverage. This approach brings the power of LLMs into software validation, making it easier to catch subtle API bugs without requiring manual test oracle definitions. The work was accepted at IEEE COMPSAC 2026.