Statistical Confidence in Functional Correctness: An Approach for AI Product Functional Correctness Evaluation

A team of researchers including Wallace Albertini and Marcos Kalinowski has introduced a novel framework called Statistical Confidence in Functional Correctness (SCFC) to address a critical gap in AI quality assessment. Published in arXiv and accepted for CAIN 2026, the approach responds to the limitations of current standards like ISO/IEC 25059, which lack practical, statistically robust methods for evaluating AI functional correctness.

The SCFC methodology consists of four key steps: defining quantitative specification limits based on business requirements, performing stratified and probabilistic sampling of test cases, applying bootstrapping techniques to estimate confidence intervals for performance metrics, and calculating a capability index as a final reliability indicator. This moves evaluation from simple point estimates (like '95% accuracy') to statements of statistical confidence that account for both average performance and variability.

In a case study involving two real-world industrial AI systems, researchers interviewed AI experts who reported the approach was feasible, valuable, and something they intended to adopt. The framework addresses the fundamental challenge of assessing probabilistic AI systems where traditional deterministic testing falls short. For enterprises deploying mission-critical AI in healthcare, finance, or autonomous systems, this provides a more rigorous way to validate that AI products meet functional requirements before deployment.