Evidence Sufficiency Under Delayed Ground Truth: Proxy Monitoring for Risk Decision Systems

A new research paper by Oleg Solozobov tackles a critical but often overlooked problem in operational AI: the 'delayed ground truth' period. In systems for fraud detection, credit scoring, and clinical risk, the final correct answer (the 'ground truth' label) can arrive days or months after the AI makes its initial decision. During this 'blind period,' the evidence used to govern and trust the AI system degrades, but current monitoring tools fail to track this decay. Solozobov's work formalizes this problem by defining 'evidence sufficiency' across four dimensions: completeness, freshness, reliability, and representativeness. It introduces a model that quantifies how label latency degrades this sufficiency, mapping different types of AI model drift (like covariate or concept drift) to specific degradation patterns.

To address the lack of labels, the paper proposes a complementary framework of seven proxy indicator categories—such as feature stability and prediction distribution—that can estimate sufficiency degradation in real-time. The system was evaluated on the large IEEE-CIS Fraud Detection dataset (~590,000 transactions) with controlled drift injection. Results showed the composite proxy monitoring detected covariate and mixed drift with a 100% detection rate. However, it confirmed a theoretical limitation: pure concept drift without feature changes remains undetectable without labels. A key finding from blind period simulation was that concept drift degrades evidence sufficiency the fastest, dropping to a score of 0.242 by day 60 compared to 0.418 for a stable model.

The framework's primary contribution is as a practical governance instrument. It translates technical drift signals into auditable 'sufficiency assessments' for regulators and risk officers, explicitly characterizing what the system can and cannot see. Accompanying the paper are open-source tools: an 'Evidence Sufficiency Calculator' and a 'Governance Drift Toolkit' to help teams implement the methodology. While mapping specific sufficiency scores to actions like model retraining requires further calibration, this work provides a much-needed structured approach to managing AI risk during the most vulnerable operational phase.