OpenAPI spec faults secretly break black-box API testing tools
Researchers inject 6 fault types into API specs, revealing hidden degradation...
Deep Dive
A new study introduces a taxonomy of six OpenAPI specification fault classes and injects them at five severity levels into two microservice benchmarks. Using three testing tools (EvoMaster, RESTler, Schemathesis), the findings show that faults in method semantics cause broad degradation across all metrics, while schema constraint relaxations create hidden quality drops not captured by code or specification coverage. The results demonstrate that specification quality directly shapes black-box API testing effectiveness.
Key Points
- Researchers introduced a taxonomy of 6 OpenAPI fault classes (method semantics, response codes, schema constraints, etc.) and tested at 5 severity levels.
- Faults in method semantics caused broad degradation across all four metrics; response code faults were weak.
- Schema constraint relaxations showed no impact on code/spec coverage but large drops in request/response quality, revealing a blind spot in coverage-only evaluations.
Why It Matters
Proves OpenAPI spec quality directly skews testing tool results, urging developers to use richer evaluation metrics beyond coverage.