Correcting Performance Estimation Bias in Imbalanced Classification with Minority Subconcepts

A team of researchers from American University and the National Research Council Canada has published a paper introducing predicted-weighted balanced accuracy (pBA), a new evaluation metric designed to correct performance estimation bias in imbalanced classification tasks where minority subconcepts exist within classes. The work, led by Taylor Maxson, Roberto Corizzo, Yaning Wu, Nathalie Japkowicz, and Colin Bellinger, addresses a critical flaw in standard class-level evaluation: models that perform well on average can fail catastrophically on specific subpopulations, yet common metrics like balanced accuracy remain misleading because they favor larger minority subgroups.

The core innovation is pBA, which replaces unavailable true subconcept labels with predicted posterior probabilities from a multiclass subconcept model. This yields a soft, uncertainty-aware weighting scheme that provides more stable and interpretable assessments when subconcept distributions are uneven but not pathological. The team validated pBA on tabular benchmarks, medical imaging datasets, and text datasets, demonstrating that unweighted scores can be highly misleading under within-class heterogeneity. The code is publicly available on GitHub, making this practical for ML practitioners working on fairness-sensitive applications.