AdaStop framework finds 84% of DNN faults using 31% of labeling budget
New cost-aware algorithm stops DNN testing when marginal benefit drops below cost.
Testing deep neural networks (DNNs) typically involves selecting inputs under a fixed labeling budget, but choosing that budget is a guessing game: too little misses faults, too much wastes resources. Bonan Shen, Wei-Jung Huang, Xin Liu, Jiazhou Gao, and Tao Ning introduce AdaStop, a framework that turns DNN testing into a cost-benefit optimization problem. Each labeled input costs $c$ and each discovered fault yields value $v$. AdaStop estimates the marginal fault discovery rate in real time and halts labeling when that rate drops below $\tau = c/v$ — the point where continuing costs more than the fault is worth.
Experiments across multiple datasets, architectures, and selection strategies show AdaStop discovers 65–84% of faults using only 9–31% of the full labeling budget. This means practitioners can confidently stop testing early, knowing that further labeling would yield diminishing returns. The framework is model-agnostic and compatible with existing test-selection methods. By explicitly modeling costs and benefits, AdaStop provides a principled, tunable stopping rule that eliminates waste and maximizes the value of each dollar spent on DNN validation.
- AdaStop models DNN testing as a cost-benefit problem: label cost $c$ vs. fault value $v$, stops when marginal rate falls below $\tau=c/v$.
- Across experiments, AdaStop discovers 65–84% of faults using only 9–31% of the labeling budget.
- Framework is model-agnostic and works with any existing test-selection strategy (e.g., coverage-guided, uncertainty-based).
Why It Matters
Cuts DNN validation costs by up to 91%, making high-assurance AI testing practical for budget-constrained teams.