CDR framework restores missing survey data with TVD < 0.001
Missing urban/rural and gender data? New AI method recovers it without breaking population statistics.
Social surveys like CHNS, NHANES, and BRFSS are foundational for population health and inequality research, but critical metadata fields (urban/rural status, gender) are often incomplete. Existing classifiers like XGBoost can impute missing values with high accuracy, but they violate the known population distribution (e.g., a 70/30 urban split becomes 55/45), introducing systemic bias. The Context Distribution Restoration (CDR) framework, introduced by Lei Zhao and collaborators, addresses this by formalizing the problem as recovering sample-level metadata assignments while respecting a known prior distribution P(M) over metadata categories.
CDR's key innovation is a recoverability-adaptive transport mechanism within an optimal transport framework. It defines recoverability theoretically as mutual information I(X; M) and approximates it via calibrated predictive uncertainty. This allows the method to dynamically adjust the trade-off between individual evidence (from covariates X) and population constraints. On three large-scale surveys (CHNS, NHANES, BRFSS) with up to 67,000 test samples, CDR achieved total variation distance (TVD) below 0.001 while incurring minimal accuracy loss—compared to XGBoost's TVD of approximately 0.11. A CHNS case study further demonstrated interpretable continuum structure, showing that CDR can handle varying recoverability across respondents.
- Unconstrained classifiers (XGBoost) yield TVD ~0.11 when imputing missing metadata, violating known population distributions.
- CDR reduces TVD to <0.001 on three large surveys (CHNS, NHANES, BRFSS) with up to 67k samples, with minimal accuracy loss.
- The method uses mutual information I(X; M) to measure recoverability per case, enabling a dynamic balance between individual signals and population priors.
Why It Matters
Enables accurate demographic data restoration for population health research without biasing survey distributions.