MASTOR Multi-Agent AI Generates 10K API Test Oracles, Beats Baselines by 30–49%

Existing automated RESTful API testing tools rely on shallow checks like HTTP status codes and schema validation, missing semantic faults, business logic violations, and state-dependent inconsistencies. To solve this, a team from multiple universities introduces MASTOR (Multi-Agent for Semantic Test Oracles), which leverages large language model agents to generate rich, executable oracles directly from API source code.

MASTOR operates in two phases. First, a source extraction agent constructs a context for each endpoint by analyzing the transitive import closure of relevant source files. Then, two parallel oracle-generation paths run over these contexts: a single-operation path produces status and field oracles per API call, while a multi-operation path generates behavioral consistency oracles for sequences of calls by capturing cross-operation semantic associations. Both paths undergo a challenger-agent review, where a dedicated reviewer identifies weaknesses and issues improvement hints, followed by normalization that filters out structurally invalid oracles.

The team evaluated MASTOR on 13 open-source RESTful API projects from the WFD and PRAB datasets, covering 296 operations and over 251,000 lines of code. MASTOR generated 10,022 oracles and achieved an average mutation score of 75.4%. In a head-to-head comparison on 50 selected operations, MASTOR’s mutation score of 69.9% dwarfed Direct Prompting’s 39.8% and SATORI’s 20.5%. The generated oracles were translated into executable assertions via ToJUnit and ToPostmanAssertify, as well as into human-readable descriptions via ToReadable, demonstrating practical integration into existing testing workflows. This approach promises to dramatically improve automated API testing by catching deep semantic bugs that current tools overlook.